Nightly Builds

Bugs affecting more than one service commonly surface only when the whole system is running. Our continuous integration system builds and runs a suite of integration tests against Spinnaker nightly on real cloud provider infrastructure to detect these bugs.

Access the CI system here:

Viewers must be a member of the build-cops GitHub Team.

Build Cop

The build cop responsibilities include:

  • Triage integration test failures on master and the 3 most recent release branches
  • Clean up orphaned resources across target cloud providers
  • Route new GitHub issues to the appropriate SIG (applying GitHub labels as appropriate). You can find the full list of SIGs in the governance repo
  • Observe any systemic problems raised in the #general and #dev Slack channels
  • Log observations and corrective actions taken in the rotation log

Process Structure

The CI system comprises both jobs, which do a specific task, and flows, which invoke a series of jobs.

Flow_BuildAndValidate is the primary entry point for the master branch flow.

Flow_BuildAndValidate_<version> is the the entry point for the respective top-level release. It is a copy of the primary Flow_BuildAndValidate when that release was cut. Top-level release flows work off their respective release-1.ABC.x branches.

As its name implies, Flow_BuildAndValidate builds and tests the whole Spinnaker system. It follows this general process:

1. Build_PrimaryArtifacts

  1. git checkout all services
  2. Constructs a BOM from the most recent commit on the target branch
  3. Builds a Docker container and a Debian package of each Spinnaker microservice.
  4. Builds additional supporting artifacts:
    • halyard
    • spin-cli
    • Changelog
  5. Publishes the BOM under the following names:
    • With the floating tag: <branchName>-latest-unvalidated (e.g. master-latest-unvalidated)
    • With a fixed tag: <branchName>-<timestamp> (e.g. master-20191213154039)
  6. Publishes the changelog

2. Validate_BomAndReportMultiPlatform

For uninteresting reasons, this job must wrap the following ValidateBomMultiPlatform in order to aggregate its results.

3. ValidateBomMultiPlatform

This “Multi-configuration project” specifies the same test(s) to run across different environments. This confirms Spinnaker works whether deployed as a single VM or in a Kubernetes cluster, for instance.

  1. Starts Halyard in a new VM
  2. Connects to this instance and executes a series of hal config steps, including account setup for the managed cloud provider(s).
  3. Deploys the configuration with hal deploy apply.
  4. Invokes citest integration tests against the new Spinnaker instance.
    • citest invokes a command to Spinnaker, and then uses the underlying cloud provider’s CLI to confirm the expected changes were made. For example, using gcloud to confirm a GCE server group was created or deleted.

Cleaning Orphaned Resources

Occasionally, integration tests fail in a way that is either undesirable or difficult to automatically clean up. Build cops should periodically ensure these orphaned resources are deleted from the following locations:

Deleting Obsolete Artifacts

The following jobs assist in removing old artifacts created during the build process: