If your Spinnaker deployment has API clients interacting with it, enabling and knowing how to operate the API rate limiter can help keep Spinnaker reliable through spikes of heavy traffic or rogue clients.
The API Rate Limiter currently supports limiting individual authenticated and anonymous principals, bucketing requests into windows that are refreshed every interval as well as global & per-principal learning mode. Both the capacity, as well as the window size, are configurable globally and per-principal.
Some metrics are generated by the rate limiter:
rateLimit.throttling: Counter. Number of enforced throttled requests.
rateLimit.throttlingLearning: Counter. Number of un-enforced throttled requests.
rateLimit.principal.throttled: Counter. A tagged metric recording the number of throttled requests per-principal. Tag key is
rateLimit.principal.remaining: Gauge. A tagged metric recording the number of remaining requests per-principal. Tag key is
controller.invocations, these metrics can give you a good sense
of your on-going rate limiting configuration.
In the case of failures (Redis unavailable) or configuration errors (principal included in both enforcing and ignoring lists), a request will gracefully fallback to learning mode.
Enabling in gate.yml
The rate limiter requires Redis to be available for Gate.
rateLimit: enabled: true learning: true redis: enabled: true # The rate (in seconds) that the bucket will be replenished rateSeconds: 10 # The number of allowed requests in the window of rateSeconds capacity: 500 # Specific principals can be given different capacities capacityByPrincipal: - principal: anonymous override: 1000 # Similarly, rateSeconds can be overridden per-principal rateSecondsByPrincipal: - principal: anonymous override: 5 # A list of principals that are being enforced. Handy for cases where you want # to incrementally enable the rate limiter enforcing: - [email protected] # A list of principals that are in learning mode. This can be useful if you # want to give some principals unlimited power. Reconsider doing this :) ignoring: - [email protected]
The rate limiter exposes four HTTP Headers on each request that a client can ingest to play nicely with the rate limiter:
X-RateLimit-Capacity: Int. Total capacity assigned to the principal.
X-RateLimit-Remaining: Int. Total number of requests left before reset.
X-RateLimit-Reset: Date. When the bucket will be replenished.
X-RateLimit-Learning: Bool. Whether or not the rate limiter is enforcing the principal.
If your Spinnaker deployment allows anonymous API access, by default all
requests will be bucketed into the same rate limit window. With a little trust,
you can allow your users to assign themselves their own anonymous rate limit
bucket by sending a
X-RateLimit-App header, where the value is the name of
Re-deploying gate to respond to real-world events is non-ideal. Redis has the capability to override any configuration defined by the static configuration. First, a list of key patterns:
rateLimit*- All rate limiter-related keys.
rateLimit:*- Principal rate limiter.
rateLimit:learning- Global learning flag.
rateLimit:enforcing- A list of principals being enforced.
rateLimit:ignoring- A list of principals in learning mode.
rateLimit:capacity:*- Per-principal capacity.
rateLimit:rateSeconds:*- Per-principal rateSeconds.
$ redis-cli # Set a new capacity 127.0.0.1:6379> set rateLimit:[email protected]:capacity 2000 # Add the [email protected] principal to the enforcing list 127.0.0.1:6379> sadd rateLimit:enforcing [email protected] (integer) 1
Code backing the rate limiter can be found in the gate repository.