{"__v":5,"_id":"56cceaacd98d851d00c0c376","category":{"__v":14,"_id":"55c6bec2b9aa4e0d0016c2c7","pages":["55c6bec3b9aa4e0d0016c2c9","55c8f451b79cb30d005b3e49","55ccea804906d12f001226e4","56c14ca9f203270d00d6c546","56c3bbeb2d97560d00e23ccf","56cce248ca43550b002814db","56cceaacd98d851d00c0c376","56cceefc723ad71d00cae46d","56ccef4e8fa8b01b00b8200e","56cd03e8723ad71d00cae4af","56cd03f2723ad71d00cae4b1","56cd061eca43550b0028151c","56cd07e4d98d851d00c0c3ba","56df35fbef8c1b320047c2d5"],"project":"55c6bec1b9aa4e0d0016c2c3","version":"55c6bec1b9aa4e0d0016c2c6","sync":{"url":"","isSync":false},"reference":false,"createdAt":"2015-08-09T02:45:22.223Z","from_sync":false,"order":0,"slug":"documentation","title":"Guides"},"parentDoc":null,"project":"55c6bec1b9aa4e0d0016c2c3","user":"55c8f42fb79cb30d005b3e48","version":{"__v":8,"_id":"55c6bec1b9aa4e0d0016c2c6","project":"55c6bec1b9aa4e0d0016c2c3","createdAt":"2015-08-09T02:45:21.683Z","releaseDate":"2015-08-09T02:45:21.683Z","categories":["55c6bec2b9aa4e0d0016c2c7","56c14bc5826df10d00e82230","56cceed8723ad71d00cae46c","56ccf29a431ada1f00e85aae","56ccf3c28fa8b01b00b82018","56ce1e6ee538330b0021ac5d","56f97e9a4c612020008f2eaf","5734fafd146eb82000597261"],"is_deprecated":false,"is_hidden":false,"is_beta":false,"is_stable":true,"codename":"","version_clean":"1.0.0","version":"1.0"},"updates":["5748799e919ad20e00b6c6fa","57576bf0c60a290e0098a6c5","575cd792443f170e00567637","583470368d454c2f0089b146"],"next":{"pages":[],"description":""},"createdAt":"2016-02-23T23:26:36.512Z","link_external":false,"link_url":"","githubsync":"","sync_unique":"","hidden":false,"api":{"results":{"codes":[]},"settings":"","auth":"required","params":[],"url":""},"isReference":false,"order":1,"body":"## General Troubleshooting Procedures\nThe following steps show a typical set of diagnostic instructions for tracking down Spinnaker issues. \n\nLet's say you have a screen in Spinnaker that is misbehaving. Here, we see that when I click on create application, the screen is stuck in an endless spinner. \n[block:image]\n{\n  \"images\": [\n    {\n      \"image\": [\n        \"https://files.readme.io/SInRAP9FTYSga0HfxwKm_stuckApplication.png\",\n        \"stuckApplication.png\",\n        \"640\",\n        \"330\",\n        \"#4484bc\",\n        \"\"\n      ]\n    }\n  ]\n}\n[/block]\n### Check browser developer console ###\nThe first thing to always do is to check the browser developer console. This will usually give you a clue to the type of issue you are seeing. In Chrome, you can access this via `View > Developer > Developer Tools`.\n[block:image]\n{\n  \"images\": [\n    {\n      \"image\": [\n        \"https://files.readme.io/3DWdo0fRSbKC6qiJZ0Rh_connectionIssue.png\",\n        \"connectionIssue.png\",\n        \"2032\",\n        \"342\",\n        \"#3c7bd9\",\n        \"\"\n      ]\n    }\n  ]\n}\n[/block]\nStart with the Console tab to see if there are any JavaScript errors that are causing grief. Any errors here would be in the Deck service.  \n\nNetwork tab and see if there are any connectivity issues or any services returning a non-200 http response. \n\nThe following table shows a list of services and the ports they are traditionally bound to in the Spinnaker configuration files. \n\n| Service | Port |   \n|---------|------|\n|Deck| 9000|\n|Clouddriver  |7002  |\n|Echo  |8089  | \n|Front50  | 8080  |\n|Gate|8084|\n|Igor|8088|\n|Orca|8083|\n|Rosco|8087|\n\n### Check Service Logs ###\n\nIf you have identified a service, you can check the service logs.\n\nIf you used one of the pre-baked Spinnaker machine images or installed Spinnaker from the .deb files, each subsystem will write its logs to:\n\n`/var/log/spinnaker/{service}/{service}.log`\n\nFor example:\n\n`/var/log/spinnaker/clouddriver/clouddriver.log`\n\n`/var/log/spinnaker/orca/orca.log`\n\n`/var/log/spinnaker/rosco/rosco.log`\n\n### Check Spinnaker API ###\n\nSpinnaker provides a swagger endpoint available through Gate (8084). All of the Spinnaker UI goes through this API. The endpoint is useful if you wish to try passing different variables to your requests to test out variables and isolate problems.\n\nTo access the API, go to http://{gate service url}/swagger-ui.html ( for example [http://localhost:8084/swagger-ui.html](http://localhost:8084/swagger-ui.html) )\n\n[block:image]\n{\n  \"images\": [\n    {\n      \"image\": [\n        \"https://files.readme.io/ADJYpWvFSOCJXODXOmCG_api.png\",\n        \"api.png\",\n        \"866\",\n        \"600\",\n        \"#2f5676\",\n        \"\"\n      ]\n    }\n  ]\n}\n[/block]\n###Check Health / Config endpoints###\n\nMost Spinnaker services have a health check endpoint configured. Sometimes this can provide useful information in terms of visualizing service health. To see if there is any information available, click on http://{service url}/health (e.g, [http://localhost:8084/health](http://localhost:8084/health)).\n\nSometimes issues arise from incorrectly set environment variables or configuration files. You can see configurations available by clicking on http://{service url}/env (e.g, [http://localhost:8084/env](http://localhost:8084/env)), which will provide a dump of the service environment. \n\nThere are a few other endpoints available. A list is available on the [Spring Boot Actuator Endpoints documentation](http://docs.spring.io/spring-boot/docs/current-SNAPSHOT/reference/htmlsingle/#production-ready).\n\n## I can't create an Application.\nThis can manifest as either an endless spinner or as an error message rendered at the bottom of the Create Application dialog.\n\nThe Spinnaker service responsible for creating applications is [front50](https://github.com/spinnaker/front50). It \"creates\" an application by adding a row to Cassandra. The first place to look is in `/var/log/spinnaker/front50/front50.log`. If you see a bunch of stack traces with references to `astyanax`, we're on the right track. The problem is that when Cassandra is upgraded, it can sometimes disable the thrift server. So we're going to first see if Cassandra is available at all, and then we'll check if thrift is enabled.\n\n1. Check if Cassandra is available via `cqlsh`. If you can connect to the cluster, Cassandra is installed and available.\n\n1. Check if Cassandra has thrift enabled via `curl localhost:9160`. If you get a connection refused, thrift is not enabled (as opposed to an 'empty reply').\n\n1. Enable thrift via this command: `nodetool enablethrift`.\n\n1. Make this setting durable by editing `/etc/cassandra/cassandra.yaml`. Find the `start_rpc` flag and set it to `true`.\n\nThrift should now be enabled. Execute `curl localhost:9160` and verify that you receive an 'empty reply'.\n\nThe last step is to restart the three Spinnaker services that require Cassandra to be available on startup: `sudo service front50 restart`, `sudo service echo restart` and `sudo service rush restart`.\n\nWe will be making [front50](https://github.com/spinnaker/front50), [echo](https://github.com/spinnaker/echo) and [rush](https://github.com/spinnaker/rush) more tolerant of an unavailable or misconfigured Cassandra cluster on startup shortly.\n\n## I changed my configuration. How do I get Spinnaker to pick up the modified configuration?\n*Note: This section is useful mainly for operators who either used one of the pre-baked Spinnaker machine images or installed Spinnaker from the .deb files (usually on an AWS or GCE VM). If doing development locally, you can probably skip this section.*\n\nThere are various ways you can modify your configuration:\n\n* Re-running `InstallSpinnaker.sh`\n* Editing `/etc/default/spinnaker`\n* Editing one of the `.yml` files (e.g. `/opt/spinnaker/config/spinnaker-local.yml`, `/opt/spinnaker/config/clouddriver.yml`, `/opt/rosco/config/rosco.yml`)\n* Modifying environment variables\n* Modifying `~/.aws/credentials`\n\nIf you've modified your configuration via any of those methods, the simplest way to have Spinnaker synchronize your configuration is to run these two commands:\n\n**1. Restart all Spinnaker services**\n\n`sudo restart spinnaker`\n\n**2. Update Deck (the browser application) settings** \n\n`sudo /opt/spinnaker/bin/reconfigure_spinnaker.sh`\n\n**3. Refresh Spinnaker Browser Cache** \n\nNavigate to the config tab of an application, ( e.g. http://localhost:9000/#/applications/{my app}/config. )\n\nYou should see a section with the heading Cache Management:\n\n[block:image]\n{\n  \"images\": [\n    {\n      \"image\": [\n        \"https://files.readme.io/1LYm2ExQoaJJyEKe12cg_cache.png\",\n        \"cache.png\",\n        \"1220\",\n        \"560\",\n        \"#67a3af\",\n        \"\"\n      ]\n    }\n  ]\n}\n[/block]\nClick on 'Refresh all caches' to make sure your configuration changes are loaded. \n\n**Additional Options: Restarting Individual Services**\n\nYou can also restart individual Spinnaker services:\n\n`sudo service restart {service-name}`\n\nFor example, the two services that typically need to be restarted to pick up account-related changes can be restarted with these commands:\n\n`sudo service restart clouddriver`\n\n`sudo service restart rosco`\n\n**Additional Options: Reload Clouddriver Config Without Restart**\n\nClouddriver also exposes an entrypoint that can be used to refresh its account lists dynamically:\n\n`curl -X POST localhost:7002/config-refresh`\n\nBut for the sake of simplicity and repeatability, the safest path is usually the coarse-grained `sudo restart spinnaker`.\n\n## Why can't I access Spinnaker using my machine's IP addr or hostname?\n*Note: This section is useful mainly for operators who either used one of the pre-baked Spinnaker machine images or installed Spinnaker from the .deb files (usually on an AWS or GCE VM). If doing development locally, you can probably skip this section.*\n\nThere is no authentication on the services so they are currently bound to localhost (since the services have admin access to your environment).\n\nA few ways you can remove this restriction:\n\n1. Edit `/etc/apache/ports.conf` and change `Listen 127.0.0.1:9000` to `Listen 9000`\n1. Add a reverse proxy for ports you wish to open in Apache\n1. Edit config files in `/opt/spinnaker/conf` to disable the bind to `localhost`. Change `localhost` to domain name or `0.0.0.0`\n\n## I don't see my VPCs in any of the dropdowns.\nSpinnaker uses naming conventions to parse a lot of things, including VPC and subnet names. If you are starting with a new VPC, we strongly suggest you name your subnets with the following pattern: \n`{vpcName}.{subnetPurpose (e.g. \"internal\")}.{availabilityZone}`\nSpinnaker will parse the subnet by splitting the name on the dots (`.`).\n\n[block:image]\n{\n  \"images\": [\n    {\n      \"image\": [\n        \"https://files.readme.io/fwTH6zttQYGBQ0mfpe1s_subnetNamingPreferred.png\",\n        \"subnetNamingPreferred.png\",\n        \"1370\",\n        \"460\",\n        \"#365069\",\n        \"\"\n      ]\n    }\n  ]\n}\n[/block]\nIf you're working with existing VPCs and subnets, there is an alternative approach to associating your subnets: add a new tag, using `immutable_metadata` as the key, with the following JSON structure:\n`{\"purpose\": \"{subnet purpose}\"}`\n\n[block:image]\n{\n  \"images\": [\n    {\n      \"image\": [\n        \"https://files.readme.io/qTEe8jmlSuS9XjP6jsea_subnetNamingAlt.png\",\n        \"subnetNamingAlt.png\",\n        \"1370\",\n        \"516\",\n        \"#34516c\",\n        \"\"\n      ]\n    }\n  ]\n}\n[/block]\nIf the `immutable_metadata` tag exists and it includes the `purpose` field, Spinnaker will use that value; it will *not* attempt to parse the name tag.\n\nIf you modify the subnet tags, you may need to refresh the browser's internal cache of the VPCs/subnets before they appear as options in dropdowns. See \"[I changed my configuration...](#i-changed-my-configuration-how-do-i-get-spinnaker-to-pick-up-the-modified-configuration)\" for instructions on refreshing the cache.\n\n## My GKE (Google Container Engine) Cluster isn't showing up as a Kubernetes account.\n\nIf you fetched your cluster's credentials with \n\n    gcloud container clusters get-credentials $CLUSTER_NAME\n    \nit's possible the credentials were downloaded in a new format not yet supported by the public client libraries. To fix this, run\n\n    gcloud config set container/use_client_certificate true\n    \nNow all future calls to get cluster credentials will download client certificates, which Spinnaker's client library can interpret.","excerpt":"","slug":"troubleshooting-guide","type":"basic","title":"Troubleshooting Guide"}

Troubleshooting Guide


## General Troubleshooting Procedures The following steps show a typical set of diagnostic instructions for tracking down Spinnaker issues. Let's say you have a screen in Spinnaker that is misbehaving. Here, we see that when I click on create application, the screen is stuck in an endless spinner. [block:image] { "images": [ { "image": [ "https://files.readme.io/SInRAP9FTYSga0HfxwKm_stuckApplication.png", "stuckApplication.png", "640", "330", "#4484bc", "" ] } ] } [/block] ### Check browser developer console ### The first thing to always do is to check the browser developer console. This will usually give you a clue to the type of issue you are seeing. In Chrome, you can access this via `View > Developer > Developer Tools`. [block:image] { "images": [ { "image": [ "https://files.readme.io/3DWdo0fRSbKC6qiJZ0Rh_connectionIssue.png", "connectionIssue.png", "2032", "342", "#3c7bd9", "" ] } ] } [/block] Start with the Console tab to see if there are any JavaScript errors that are causing grief. Any errors here would be in the Deck service. Network tab and see if there are any connectivity issues or any services returning a non-200 http response. The following table shows a list of services and the ports they are traditionally bound to in the Spinnaker configuration files. | Service | Port | |---------|------| |Deck| 9000| |Clouddriver |7002 | |Echo |8089 | |Front50 | 8080 | |Gate|8084| |Igor|8088| |Orca|8083| |Rosco|8087| ### Check Service Logs ### If you have identified a service, you can check the service logs. If you used one of the pre-baked Spinnaker machine images or installed Spinnaker from the .deb files, each subsystem will write its logs to: `/var/log/spinnaker/{service}/{service}.log` For example: `/var/log/spinnaker/clouddriver/clouddriver.log` `/var/log/spinnaker/orca/orca.log` `/var/log/spinnaker/rosco/rosco.log` ### Check Spinnaker API ### Spinnaker provides a swagger endpoint available through Gate (8084). All of the Spinnaker UI goes through this API. The endpoint is useful if you wish to try passing different variables to your requests to test out variables and isolate problems. To access the API, go to http://{gate service url}/swagger-ui.html ( for example [http://localhost:8084/swagger-ui.html](http://localhost:8084/swagger-ui.html) ) [block:image] { "images": [ { "image": [ "https://files.readme.io/ADJYpWvFSOCJXODXOmCG_api.png", "api.png", "866", "600", "#2f5676", "" ] } ] } [/block] ###Check Health / Config endpoints### Most Spinnaker services have a health check endpoint configured. Sometimes this can provide useful information in terms of visualizing service health. To see if there is any information available, click on http://{service url}/health (e.g, [http://localhost:8084/health](http://localhost:8084/health)). Sometimes issues arise from incorrectly set environment variables or configuration files. You can see configurations available by clicking on http://{service url}/env (e.g, [http://localhost:8084/env](http://localhost:8084/env)), which will provide a dump of the service environment. There are a few other endpoints available. A list is available on the [Spring Boot Actuator Endpoints documentation](http://docs.spring.io/spring-boot/docs/current-SNAPSHOT/reference/htmlsingle/#production-ready). ## I can't create an Application. This can manifest as either an endless spinner or as an error message rendered at the bottom of the Create Application dialog. The Spinnaker service responsible for creating applications is [front50](https://github.com/spinnaker/front50). It "creates" an application by adding a row to Cassandra. The first place to look is in `/var/log/spinnaker/front50/front50.log`. If you see a bunch of stack traces with references to `astyanax`, we're on the right track. The problem is that when Cassandra is upgraded, it can sometimes disable the thrift server. So we're going to first see if Cassandra is available at all, and then we'll check if thrift is enabled. 1. Check if Cassandra is available via `cqlsh`. If you can connect to the cluster, Cassandra is installed and available. 1. Check if Cassandra has thrift enabled via `curl localhost:9160`. If you get a connection refused, thrift is not enabled (as opposed to an 'empty reply'). 1. Enable thrift via this command: `nodetool enablethrift`. 1. Make this setting durable by editing `/etc/cassandra/cassandra.yaml`. Find the `start_rpc` flag and set it to `true`. Thrift should now be enabled. Execute `curl localhost:9160` and verify that you receive an 'empty reply'. The last step is to restart the three Spinnaker services that require Cassandra to be available on startup: `sudo service front50 restart`, `sudo service echo restart` and `sudo service rush restart`. We will be making [front50](https://github.com/spinnaker/front50), [echo](https://github.com/spinnaker/echo) and [rush](https://github.com/spinnaker/rush) more tolerant of an unavailable or misconfigured Cassandra cluster on startup shortly. ## I changed my configuration. How do I get Spinnaker to pick up the modified configuration? *Note: This section is useful mainly for operators who either used one of the pre-baked Spinnaker machine images or installed Spinnaker from the .deb files (usually on an AWS or GCE VM). If doing development locally, you can probably skip this section.* There are various ways you can modify your configuration: * Re-running `InstallSpinnaker.sh` * Editing `/etc/default/spinnaker` * Editing one of the `.yml` files (e.g. `/opt/spinnaker/config/spinnaker-local.yml`, `/opt/spinnaker/config/clouddriver.yml`, `/opt/rosco/config/rosco.yml`) * Modifying environment variables * Modifying `~/.aws/credentials` If you've modified your configuration via any of those methods, the simplest way to have Spinnaker synchronize your configuration is to run these two commands: **1. Restart all Spinnaker services** `sudo restart spinnaker` **2. Update Deck (the browser application) settings** `sudo /opt/spinnaker/bin/reconfigure_spinnaker.sh` **3. Refresh Spinnaker Browser Cache** Navigate to the config tab of an application, ( e.g. http://localhost:9000/#/applications/{my app}/config. ) You should see a section with the heading Cache Management: [block:image] { "images": [ { "image": [ "https://files.readme.io/1LYm2ExQoaJJyEKe12cg_cache.png", "cache.png", "1220", "560", "#67a3af", "" ] } ] } [/block] Click on 'Refresh all caches' to make sure your configuration changes are loaded. **Additional Options: Restarting Individual Services** You can also restart individual Spinnaker services: `sudo service restart {service-name}` For example, the two services that typically need to be restarted to pick up account-related changes can be restarted with these commands: `sudo service restart clouddriver` `sudo service restart rosco` **Additional Options: Reload Clouddriver Config Without Restart** Clouddriver also exposes an entrypoint that can be used to refresh its account lists dynamically: `curl -X POST localhost:7002/config-refresh` But for the sake of simplicity and repeatability, the safest path is usually the coarse-grained `sudo restart spinnaker`. ## Why can't I access Spinnaker using my machine's IP addr or hostname? *Note: This section is useful mainly for operators who either used one of the pre-baked Spinnaker machine images or installed Spinnaker from the .deb files (usually on an AWS or GCE VM). If doing development locally, you can probably skip this section.* There is no authentication on the services so they are currently bound to localhost (since the services have admin access to your environment). A few ways you can remove this restriction: 1. Edit `/etc/apache/ports.conf` and change `Listen 127.0.0.1:9000` to `Listen 9000` 1. Add a reverse proxy for ports you wish to open in Apache 1. Edit config files in `/opt/spinnaker/conf` to disable the bind to `localhost`. Change `localhost` to domain name or `0.0.0.0` ## I don't see my VPCs in any of the dropdowns. Spinnaker uses naming conventions to parse a lot of things, including VPC and subnet names. If you are starting with a new VPC, we strongly suggest you name your subnets with the following pattern: `{vpcName}.{subnetPurpose (e.g. "internal")}.{availabilityZone}` Spinnaker will parse the subnet by splitting the name on the dots (`.`). [block:image] { "images": [ { "image": [ "https://files.readme.io/fwTH6zttQYGBQ0mfpe1s_subnetNamingPreferred.png", "subnetNamingPreferred.png", "1370", "460", "#365069", "" ] } ] } [/block] If you're working with existing VPCs and subnets, there is an alternative approach to associating your subnets: add a new tag, using `immutable_metadata` as the key, with the following JSON structure: `{"purpose": "{subnet purpose}"}` [block:image] { "images": [ { "image": [ "https://files.readme.io/qTEe8jmlSuS9XjP6jsea_subnetNamingAlt.png", "subnetNamingAlt.png", "1370", "516", "#34516c", "" ] } ] } [/block] If the `immutable_metadata` tag exists and it includes the `purpose` field, Spinnaker will use that value; it will *not* attempt to parse the name tag. If you modify the subnet tags, you may need to refresh the browser's internal cache of the VPCs/subnets before they appear as options in dropdowns. See "[I changed my configuration...](#i-changed-my-configuration-how-do-i-get-spinnaker-to-pick-up-the-modified-configuration)" for instructions on refreshing the cache. ## My GKE (Google Container Engine) Cluster isn't showing up as a Kubernetes account. If you fetched your cluster's credentials with gcloud container clusters get-credentials $CLUSTER_NAME it's possible the credentials were downloaded in a new format not yet supported by the public client libraries. To fix this, run gcloud config set container/use_client_certificate true Now all future calls to get cluster credentials will download client certificates, which Spinnaker's client library can interpret.