Custom Settings

“While Halyard handles the majority of Spinnaker configuration, there will always be feature flags and properties either too new or niche to be supported by Halyard.”

Furthermore, some users will be more comfortable forgoing Halyard’s configuration generation, or will want to port their old Spinnaker configuration into Halyard. For these users & use-cases, Halyard supports custom Profiles , and custom Service Settings .

Custom Profiles versus Service settings

The two most common ways of customizing your Spinnaker installation is through the use of either a custom profile, for example clouddriver-local.yml, or a service setting, for example clouddriver.yml. Each of these is created in separate subfolders inside the main Halyard deployment profile folder. In a default Halyard profile, this means profiles go in .hal/default/profiles/ while service settings go in .hal/default/service-settings.

Custom profiles are configuration files that override the default configuration files for each Spinnaker Service. Service-Settings are used by Halyard to determine how to build the Spinnaker deployment and contain information specific to how each Spinnaker Service will be deployed.

For example, if you want to substitute which image file will be used for a Spinnaker service, or what environment variables the service will be run with, then you’ll need to modify the service settings for that service.

To override the debugging level specified in clouddriver.yml, create a custom clouddriver-local.yml file and put it in the /profiles/ directory.

Custom Profiles

If you want to supply custom Profiles, for a particular Deployment ($DEPLOYMENT, typically default) place them in ~/.hal/$DEPLOYMENT/profiles/. The will override any profiles generated by Halyard, and will be copied over by Profile name.

For example, if you want to supply a different clouddriver.yml, you can do so by placing it in ~/.hal/$DEPLOYMENT/profiles/clouddriver.yml, and running hal deploy apply.

For any of the Spring based services, any profile matching ${SERVICENAME}-.*\.yml will be copied and applied to $SERVICENAME. Since this approach will completely overwrite the generated service, and you may want to only supply a few flags, you can instead provide them in ${SERVICENAME}-local.yml, which will always be loaded by $SERVICENAME with no further configuration needed. Further Spring Profiles can be activated by configuring their env to include SPRING_PROFILES_ACTIVE entries for your desired service. See the below Custom Service Settings for more information.

You can validate that your Custom Profiles have been picked up and applied to their services by checking ~/.hal/$DEPLOYMENT/history/service-profiles.yml.

Custom Profile for Deck

To supply custom settings or new values that override generated settings for the Deck UI, you can place them in a file called settings-local.js. For example, to enable the chaosMonkey feature flag, you can create the following file:

~/.hal/default/profiles/settings-local.js:

window.spinnakerSettings.feature.chaosMonkey = true

Custom Service Settings

For each service, $SERVICENAME, you can supply a file ~/.hal/$DEPLOYMENT/service-settings/$SERVICENAME.yml to override any settings as found in ~/.hal/$DEPLOYMENT/history/service-settings.yml. For example, if you wanted Echo to listen on port 443 with the JAVA_OPTS="-Xms256m -Xmx512m" environment variables set, you create the following file:

~/.hal/default/service-settings/echo.yml:

port: 443
env:
  JAVA_OPTS: '-Xms256m -Xmx512m'

If you wanted to specify a secure connection to redis, you would create the following file:

~/.hal/default/service-settings/redis.yml:

overrideBaseUrl: redis://someuser:somepassword@localhost:6379

Tweakable Service Settings

Below are all the Service Settings you can configure. Keep in mind the ones generated by Halyard are generally best and are all that we test, so there is no guarantee making changes to these will result in a working Spinnaker cluster.

ValueDescription
addressThe address other services will use to lookup this service; e.g. localhost or redis.service.spinnaker.consul.
artifactIdA fully-qualified identifier for the Artifact this service is to run - see your Service Settings output path for examples, since this varies by platform/deployment environment.
basicAuthEnabledWhether or not to enable HTTP basic auth for this service.
enabledWhether or not this service is enabled. Be careful, since disabling critical services like consul-client can make your entire installation unusable and un-upgradable.
envEnvironment variables to make available to this Service. Supplied as YAML key-value pairs.
healthEndpointThe HTTP endpoint that can be queried for health status. Leave empty if you want Halyard to simply check that a TCP connection is openable. This information is typically transmitted to the platform’s discovery system.
hostThe host that this service will bind to. 0.0.0.0 will always work, but more restrictive bindings may be more secure.
kubernetesYou can add Kubernetes specific service settings, see details below.
locationThe Spinnaker “location” this will be installed in. This is a namespace in Kubernetes, and a zone in GCE.
monitoredWhether or not this has monitoring endpoints exposed.
overrideBaseUrlThe baseURL this service is reachable on. This is already made configurable for Deck & Gate via hal config security, since these are both public-facing and may service from different hosts than they are discoverable on internal to Spinnaker.
portThe port number this service is bound to, and will accept requests on.
safeToUpdateWhether or not this service can be shutdown, and spun on a new VM/container. This protects datastores like Vault & Redis from being taken down from Halyard.
schemeThe URI scheme for addressing this service, e.g. http vs. https.
skipLifeCycleManagementWhether or not Halyard should skip managing a service’s life cycle.
targetSizeThe initial number of nodes this service will be created with. This is only respected on the initial deployment, and further edits will be rejected in favor of the prior service version’s size.

Kubernetes

Inside the kubernetes item at the top level you can add a number of custom service settings that impact how the Spinnaker services are deployed onto a Kubernetes cluster. This lets you partially configure how the Kubernetes Deployment, Service, Pods and other Kubernetes resources are deployed by Halyard. This allows you to do things like add a Kubernetes secret or emptydir to be mounted as a Volume and VolumeMount, change the default Kubernetes deployment strategy and more.

At a high level, the configurable items inside the Kubernetes service settings are:

ItemDetails
deploymentStrategyDefines either RollingUpdate or Recreate deployment strategy, including setting MaxSurge and MaxUnavailable See Deployment Strategy below.
imagePullSecretsDefine secrets to use to pull a custom artifact when using an artifactId to swap out docker image. More details can be found in the Image Pull Secrets section .
nodePortWhen using a serviceType of NodePort, this provides the NodePort value.
nodeSelectorsProvide a list of nodeSelectors key-value pairs to add to the pod specification. See Node Selectors below.
affinityProvide a list of affinity rules applied to a Spinnaker component’s pod specification. See Affinity below.
podAnnotationsProvide a list of annotations to put on the deployed pods. See Annotations and Labels below.
podLabelsProvide a list of labels to put on the deployed pods. See Annotations and Labels below.
securityContextSet the securityContext that the Spinnaker services should run using in Kubernetes
serviceAnnotationsProvide a list of annotations to put on the deployed services. See Annotations and Labels below.
serviceLabelsProvide a list of labels to put on the deployed services. See Annotations and Labels below.
serviceAccountNameProvide a default serviceAccount under which to run Spinnaker services.
serviceTypeDefine a specific serviceType for deployed services, ClusterIP (Default) or NodePort
useExecHealthCheckDisable the exec-based healthcheck if necessary. See useExecHealthCheck for details.
volumesDefine a set of Volume and VolumeMount items to be attached to the Pods through the Deployment. See Custom Volumes

imagePullSecrets

If making use of custom artifactIds from an authenticated docker registry imagePullSecrets must be made available in the replicaset definition. To specify imagePullSecrets to custom artifactIds they can be specified as follows:

kubernetes:
  imagePullSecrets:
  - desired-image-pull-secret1
  - desired-image-pull-secret2

podAnnotations, podLabels and serviceAnnotations, serviceLabels

Annotations and labels hold metadata that can be used to specify special behavior within Kubernetes. To apply annotations or labels to the Pod and Service resources for a particular Spinnaker service you can use something like:

kubernetes:
  podAnnotations:
    example/podAnnotation: spinnaker.io
    example/podAnnotation-2: halyard
  podLabels:
    examplePodLabel: examplePodLabelValue
  serviceLabels:
    exampleServiceLabel: exampleServiceLabelValue
  serviceAnnotations:
    example/serviceAnnotation: spinnaker.io
    example/serviceAnnotation-2: halyard

These are all optional ways to control what metadata elements exist on the resources deployed by Halyard. For additional information about how these can be used, you can look at the Annotations or Labels and Selectors pages from Kubernetes.

nodeSelectors

Node selector annotations will put out nodeSelector values in the Pod specification. They generally follow a similar syntax to the annotations and labels. For example:

kubernetes:
  nodeSelector:
     exampleNodeKey: exampleNodeValue
  deploymentEnvironment:
    nodeSelector:
      exampleNodeKey: exampleNodeValue

Additional information regarding Node Selectors can be found in the Kubernetes NodeSelector documentation .

affinity

Affinity annotations generate affinity values in the Pod specification and are more expressive than nodeSelector. You can set precedence rather than specify an affinity hard requirement. The two types are node affinity and inter-pod affinity/anti-affinity.

Provide configuration per spinnaker component like the below example.

kubernetes:
  deploymentEnvironment:
    affinity:
      spin-clouddriver:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: spinnaker
                operator: In
                values:
                - 'true'

See the Kubernetes affinity documentation for more information.

useExecHealthCheck

By default halyard deploys services with an exec-based health check in order to improve compatibility with Istio. This however can break functionality for implementations of Load Balancer service types and Ingress Controllers that rely on having an http health check to validate. Setting kubernetes.useExecHealthCheck: false will switch the check method to be http based for such use cases.

Using Custom Volumes

You can define a set of custom volumes for a particular service that will become Volume and VolumeMount items in the generated pod specifications when Halyard deploys Spinnaker on Kubernetes. This can be useful if you need to add custom CA Certificates or other files to Spinnaker service consistently.

The Kubernetes service settings allow you to mount volumes of type configMap, secret and emptyDir with the below syntax.

kubernetes:
  volumes:
  - id: secret-example-vol
    type: secret
    mountPath: /path/to/location/of/secret
  - id: configmap-example-vol
    type: configMap
    mountPath: /path/to/location/of/configMap
  - id: emptydir-example-vol
    type: emptyDir
    mountPath: /path/to/location/of/emptyDir

Make sure that the id field matches the name of the existing secret or configmap resource in Kubernetes that you need mounted.

Deployment Strategy

You can change the deployment strategy that Kubernetes uses when deploying Spinnaker services. For example, if you wanted to adjust the maxSurge and maxUnavailable percentages for a rolling update of Clouddriver, you would define a clouddriver.yml file in your Halyard service-settings like the one below.

kubernetes:
  deploymentStrategy:
    type: rollingUpdate
    strategySettings:
      maxSurge: 100%
      maxUnavailable: 50%

If you wanted to use a Recreate strategy instead, you would use:

kubernetes:
  deploymentStrategy:
    type: recreate

For additional information regarding the implication of each of these deployment strategies, consult the Updating a Deployment documentation.