Bugs affecting more than one service commonly surface only when the whole system is running. Our continuous integration system builds and runs a suite of integration tests against Spinnaker nightly on real cloud provider infrastructure to detect these bugs.
Access the CI system here: https://builds.spinnaker.io
Viewers must be a member of the
The build cop responsibilities include:
- Triage integration test failures on
masterand the 3 most recent release branches
- Clean up orphaned resources across target cloud providers
- Route new GitHub issues to the appropriate SIG (applying GitHub labels as appropriate). You can find the full list of SIGs in the governance repo
- Observe any systemic problems raised in the #general and #dev Slack channels
- Log observations and corrective actions taken in the rotation log
The CI system comprises both jobs, which do a specific task, and flows, which invoke a series of jobs.
Flow_BuildAndValidate is the primary entry point for the
master branch flow.
Flow_BuildAndValidate_<version> is the the entry point for the respective top-level release. It is a copy of the primary
Flow_BuildAndValidate when that release was cut. Top-level release flows work off their respective
As its name implies,
Flow_BuildAndValidate builds and tests the whole Spinnaker system. It follows this general process:
git checkoutall services
- Constructs a BOM from the most recent commit on the target branch
- Builds a Docker container and a Debian package of each Spinnaker microservice.
- Builds additional supporting artifacts:
- Publishes the BOM under the following names:
- With the floating tag:
- With a fixed tag:
- With the floating tag:
- Publishes the changelog
For uninteresting reasons, this job must wrap the following
ValidateBomMultiPlatform in order to aggregate its results.
This “Multi-configuration project” specifies the same test(s) to run across different environments. This confirms Spinnaker works whether deployed as a single VM or in a Kubernetes cluster, for instance.
- Starts Halyard in a new VM
- Connects to this instance and executes a series of
hal configsteps, including account setup for the managed cloud provider(s).
- Deploys the configuration with
hal deploy apply.
citestintegration tests against the new Spinnaker instance.
citestinvokes a command to Spinnaker, and then uses the underlying cloud provider’s CLI to confirm the expected changes were made. For example, using
gcloudto confirm a GCE server group was created or deleted.
Cleaning Orphaned Resources
Occasionally, integration tests fail in a way that is either undesirable or difficult to automatically clean up. Build cops should periodically ensure these orphaned resources are deleted from the following locations:
Deleting Obsolete Artifacts
The following jobs assist in removing old artifacts created during the build process: