At scale: per-instance state, retry storms, and coordinated shedding

Every lesson so far quietly assumed one process: one breaker, one thread pool, one counter. Production is fifty instances behind a load balancer, and the assumptions break in ways that are not obvious. Each instance has its own breaker with its own count, so instance A can be tripped while instance B still hammers the dying dependency — there is no shared verdict. Worse, you almost never have just a breaker; you have retries under it, and retries stack multiplicatively through a call chain. Google’s SRE book has the canonical horror: three layers of service, each retrying up to four times, turns one user request into 4 × 4 × 4 = 64 calls to the bottom service — a request amplifier that converts a small downstream wobble into a self-sustaining retry storm. Resilience at scale is not the single-process state machine of the last five lessons; it is a fleet property, and the fleet has failure modes a single breaker never sees.

Per-instance state: fifty breakers, no shared verdict

A circuit breaker’s state lives in the process that owns it. With fifty instances you have fifty independent breakers, each counting only the calls it made. This has direct consequences. Instance A might see enough failures to trip while instance B, having sampled a luckier slice of traffic, stays closed and keeps calling the sick dependency. There is no shared verdict — the dependency does not get “a breaker,” it gets fifty opinions. Right after a deploy or a scale-up, a fresh instance starts with an empty window and a closed breaker, so it must rediscover the outage from scratch by failing real calls, even though forty-nine of its siblings already know.

You can centralize breaker state in a shared store (Redis, a coordination service) so the fleet shares one count — but now the breaker check is a network call on the hot path, the shared store is a new dependency that can itself fail, and you have traded local-but-stale for shared-but-coupled. Most production systems keep breaker state local on purpose and accept the fuzziness, because a breaker is a statistical safety device, not a precise switch: fifty instances each sampling the dependency will, in aggregate, converge on tripping during a real outage. Local state is the pragmatic default; shared state is a deliberate, costly choice.

Retry amplification: the multiplier under the breaker

The dangerous interaction is not the breaker — it is the retry layered beneath it. Retries multiply along a call chain. If service A calls B calls C, and each retries 4× on failure, one inbound request to A can become 64 requests at C. When C is already struggling, this is the worst possible response: the layer that should reduce load instead multiplies it, and every layer’s retries feed the layer below. This is a retry storm (or retry amplification), and it is how a brief downstream blip becomes a sustained, self-inflicted overload that outlives the original fault.

The senior fixes are layered and specific:

• Retry at one level, not every level. The SRE guidance is to retry close to the failure or at the top, but not at every layer simultaneously — stacked retries are what produce the 64×. Pick a layer; make the others fail fast.
• Cap the absolute retry rate, not just the per-call count. Google’s pattern is a per-process budget — “60 retries per minute” as a server-wide cap — so retries cannot scale with traffic into a storm.
• Exponential backoff with jitter. Fixed-interval retries from many clients re-synchronize into waves; AWS’s builder’s library shows that adding randomized jitter to exponential backoff spreads the retries out and is what actually breaks the wave. Backoff alone is not enough — without jitter, every client backs off by the same amount and they all return together.
• Breakers sit above retries. The breaker is the circuit-level off-switch: once it is open, the retries beneath it never fire at all, because the whole call short-circuits. A breaker is what makes a retry policy safe — it bounds the storm.

Herding: half-open and recovery are fleet events

The half-open state from lesson two has a distributed failure mode. When fifty instances tripped at roughly the same moment, their cooldown timers also expire at roughly the same moment, so all fifty enter half-open and fire their probe calls together — a synchronized thundering herd onto a dependency that has just barely come back. The fix is the same primitive as retry backoff: jitter the probe timing so the fleet’s probes spread across a window instead of landing in one spike. AWS’s token-bucket retry throttling is the related idea on the retry side — each client maintains a token bucket that only permits retries while tokens remain, so a fleet-wide failure cannot translate into a fleet-wide retry burst.

Coordinated shedding: the floor must agree

Load shedding from the last lesson also changes shape across a fleet. If each instance sheds independently based on its own load, a load balancer routing unevenly can have some instances shedding hard while others sit idle — the fleet sheds the wrong requests. Worse, an instance that sheds a request does not make the work disappear; if the client retries, the shed request lands on another instance, so uncoordinated shedding can just shuffle load around the fleet instead of reducing it. The stable version is shedding that the fleet agrees on: shed by priority (drop low-priority traffic everywhere first, consistently), shed deterministically on a signal every instance can compute the same way, and pair it with the deadline-aware queuing from the last lesson so a request shed at the front door is not retried into the back. The principle that ties the whole lesson together: in a distributed system, every per-instance safety device — breaker, retry, shed — needs a fleet-level story, or the instances’ locally-correct decisions compose into a globally-wrong outcome.