Distributed Systems
Retry amplification: multiple-choice review
Crux Multiple-choice synthesis across the retry-amplification unit — fan-out math, metastable loops, jitter, retry budgets, and circuit breakers under load.
Your altitude — climbing toward senior
ZeroJuniorMiddleSenior
You are at senior altitude — in orbitSix questions that cut across the whole unit. Each one mirrors a call you make mid-incident — not a definition to recite, but a fan-out number to compute and a fix to rank while the backend is on fire.
Goal
Confirm you can connect retry fan-out math, the metastable sustaining loop, and the defense ladder — jitter, retry budgets, circuit breakers, idempotency, deadline propagation — the synthesis the unit built toward.
Quiz
Completed
A request crosses 5 service layers and every layer retries 3 times on failure. In the worst case, how many calls hit the deepest dependency for one failing request, and why does this ambush people?
Heads-up Retries multiply, they do not add. Each layer's retry re-runs the whole retry chain beneath it, so amplification is retries^depth (3⁵ = 243), not retries × depth (15).
Heads-up Every layer with its own retry policy contributes a factor. The leaf's 3 is just the innermost term of 3 × 3 × 3 × 3 × 3.
Heads-up Amplification is about one failing path being re-attempted at every level on its way down, not simultaneous independent failures. A single failing request alone produces 3⁵.
Quiz
Completed
The 8-second database failover that triggered the incident finished 30 minutes ago, but the fleet is still down under a retry storm. What is actually keeping it down?
Heads-up The defining property of metastable failure is that it persists after the trigger clears. Hunting the already-healed trigger is the classic time sink; the self-sustaining retry loop is the real cause.
Heads-up No new bug is required. The retry feedback loop alone (timeouts to retries to more load to more timeouts) holds the system in the degraded stable state on its own.
Heads-up Metastable failures generally do not self-heal because the load that sustains them does not drop. Operators have to shed load, open breakers, or restart tiers to force the system back to its healthy state.
Quiz
Completed
A team adds fixed-interval retries (wait 200 ms, retry, wait 200 ms, retry) at every layer to 'be gentle.' During the next dependency blip the outage is worse than before. Why?
Heads-up A longer fixed interval still re-synchronises the herd; everyone just stamps together on a slower beat. The defect is the lack of randomness (jitter), not the specific interval length.
Heads-up Jittered, budgeted retries genuinely help — most transient blips succeed on a second attempt. The problem here is unjittered fixed timing, not the existence of retries.
Heads-up The synchronized spike is self-inflicted by fixed intervals. Jitter spreads attempts and demonstrably cuts both total calls and time-to-recovery versus fixed or plain exponential backoff.
Quiz
Completed
You have already shipped exponential backoff with full jitter at every layer, yet a full dependency outage still drives the backend to roughly 80x its normal load. What did jitter not buy you, and what do you add?
Heads-up Jitter only de-correlates clients in time. It never bounds how many total retries occur — that is precisely what a retry budget is for.
Heads-up Raising the per-request limit increases attempts, not decreases them. Bounding amplification needs a fleet-wide budget (retries as a fraction of traffic), not a per-request count.
Heads-up A ~10% retry budget turns unbounded geometric growth into at most ~10% extra load even in a full outage. 80x is exactly what the budget exists to prevent.
Quiz
Completed
A circuit breaker opens after the failure rate to a dependency crosses its threshold. What does this accomplish that a retry budget alone does not?
Heads-up A breaker is not a permanent kill switch. After a cooldown it goes half-open: one probe succeeds and it closes; the probe fails and it re-opens. Recovery is automatic.
Heads-up The breaker does not touch the dependency's internals. It stops the caller from sending doomed requests, which indirectly lets the backend recover by removing retry load.
Heads-up Breaker, budget, and jittered backoff are complementary layers: jitter shapes timing, the budget caps volume, the breaker grants a zero-traffic recovery window. You want all three, not one instead of the others.
Quiz
Completed
Reviewing a service's retry config, you find it retries every failure up to 3 times — including 400s, 409 conflicts, and non-idempotent POSTs — with no deadline propagation. Which two rules is it violating, and why do they matter most under load?
Heads-up More retries on non-retryable errors and dead requests just multiplies amplification. The fix is fewer, smarter retries: only retryable errors, only within the live deadline.
Heads-up A 400 is a deterministic client error; retrying it always fails again and wastes a backend call. Only transient, retryable errors (timeouts, 503s, connection resets) should be retried.
Heads-up Without deadline propagation a layer happily retries a request whose caller already timed out — pure wasted work that produces nothing but more load, which is exactly what feeds a metastable storm.
Recap
The through-line is one decision tree: retries compose by multiplication (retries^depth, so 3⁵ = 243), and a tiny error rate becomes a self-inflicted DDoS; the storm then sustains itself as a metastable failure long after the trigger clears. The defenses layer from broadest to finest — jitter de-synchronizes the herd in time, a ~10% retry budget caps the volume, a circuit breaker grants a zero-retry recovery window — and the non-negotiable rules are: retry only idempotent operations and retryable errors, and propagate the deadline so no layer ever retries a request that is already dead.