Resilience: cascading retries, circuit breakers, and error budgets

A third-party payment API slows to 5 seconds per response. Your server times out waiting, returns a 503. The client library retries after 100 ms, 200 ms, 400 ms. So do 1,000 other users. Your origin server, which normally handles 500 req/s, is now receiving 3,000 req/s — and still waiting on the slow payment API. Within 30 seconds, the entire site is down. The payment API was slow, not broken. But you turned “slow” into “catastrophic” by retrying without coordination.

The cascade anatomy

A retry storm is a positive feedback loop: one timeout → many retries → more load → more timeouts → more retries.

The amplification math. A 99.9% origin server (1 error per 1,000 requests) normally passes traffic fine. During a slowdown, requests queue. 1,000 clients each retry 3 times with a naive backoff. Origin traffic: 3× normal. Origin slows more. More retries. In 30 seconds, origin receives 10× normal traffic. A 99.9% server is now the bottleneck for 100% of traffic. A single degraded dependency turns a 0.1% error rate into a 100% outage.

The twelve-layer problem. Each layer in the request chain has its own retry logic:

• DNS client retries on SERVFAIL (3 attempts, 2 s apart).
• TCP SYN retries on timeout (exponential backoff, up to 3 attempts).
• TLS handshake retries on timeout (1–2 attempts).
• HTTP client library retries on 5xx (3 attempts, default exponential).
• Application-layer retry (business logic retries failed service calls).

All twelve layers retry independently. A single failure at layer 6 (origin processing) causes retries at layers 3 through 12 simultaneously. In aggregate, what was one failed request becomes 30+ network operations — all compounding the load on the already-struggling origin.

Defense 1: Exponential backoff with jitter

Naive retry. Client sees 503, retries immediately. Server still broken. Client sees 503, retries again. Server never recovers.

Exponential backoff. First retry after 100 ms, second after 200 ms, third after 400 ms, fourth after 800 ms (doubling each time, capped at some max). The origin has time to recover between retries.

Why jitter is required. If 1,000 clients all hit a limit and all use the same backoff parameters, they all retry at t=100 ms, t=200 ms, etc. simultaneously. The origin gets a burst every 100 ms — a thundering herd. Jitter: add ±25% random variation to each retry delay. The 1,000 clients spread their retries across a 200 ms window. Origin load is constant instead of pulsed.

Full jitter formula (Jeff Atwood / Amazon): sleep = random_between(0, min(cap, base * 2^attempt)). This gives up to cap ms sleep on the final attempt, fully randomised — the most effective at preventing thundering herd.

Defense 2: Circuit breaker

Pattern from electrical engineering: a switch that opens when current exceeds a threshold, preventing damage from cascading.

Three states:

1. Closed (normal). Requests pass through. Error rate and latency are monitored.
2. Open (tripped). Error rate or latency exceeded threshold for the last N requests. New requests fail immediately (fast error, no network call). Server is not contacted.
3. Half-open (probe). After a cool-down window (e.g., 30 s), a single request is allowed through. If it succeeds, circuit closes (normal). If it fails, circuit reopens.

Why fast failure helps. If the circuit is open and returns 503 in <1 ms, the client fails fast. With exponential backoff + jitter, the client waits and retries. The origin gets relief. When the origin recovers, the half-open probe succeeds, and traffic resumes. Without a circuit breaker, the client keeps retrying for seconds — adding load to an already-struggling origin.

Implementation. Libraries: Netflix Hystrix (Java, deprecated), Resilience4j (Java), Polly (.NET), opossum (Node.js). Configuration: failure threshold (e.g., 50% error rate over last 10 requests), half-open cooldown (30 s), sliding window size.

Defense 3: Bulkhead

Named after ship compartments that isolate flooding to one section. In software: give each dependency a separate thread pool (or async semaphore). If the payment API is slow, it fills the payment thread pool. The user API, product API, and CDN-fetch thread pools remain available. Slow dependency does not starve all threads.

Without bulkheads. Shared thread pool of 100 threads. Payment API slows → 100 requests queue → all 100 threads occupied → every API endpoint (user, product, search) is blocked. Site is down for all features, not just payment.

With bulkheads. Payment: 20 threads. User: 30 threads. Product: 30 threads. Misc: 20 threads. Payment API slows → 20 payment threads occupied → payment endpoint slow → all other endpoints unaffected. Graceful partial degradation.

Defense 4: Load shedding

When the request queue exceeds the server’s capacity to process, shed the excess with fast 503 instead of letting requests timeout.

Why fast 503 is better than timeout. A request that eventually times out after 30 s holds a thread, a DB connection, and memory for 30 s. A request that gets 503 in <1 ms releases all resources immediately. The client can retry with backoff. The server stays stable.

Implementation. Nginx: limit_req_zone + limit_req (rate-based, returns 503 when queue is full). In application: check request queue depth; if > threshold, return 503 immediately. Combine with stale-while-revalidate at the CDN so cached content is still served from edge during origin overload.

Error budgets and SLO-driven release cadence

SLO (Service Level Objective). A 99.9% availability SLO over a 30-day window means the service is allowed 0.1% errors — about 43.2 minutes of downtime. A 99.95% SLO allows ~21.6 minutes.

Error budget. The allowed failure time. When the budget is burned:

• At 50% consumed: slow down risky changes (feature releases, config changes).
• At 100% consumed: freeze all non-essential releases until reliability improves.

This formalises the latency/feature-velocity trade-off without politics. Engineers can ship fast while they have budget; operations can enforce a release freeze when the budget is gone.

Performance budgets. Extend the same idea to page load: set explicit targets (LCP < 2.5 s, JS bundle < 200 KB). Enforce in CI with tools like bundlewatch, size-limit, Lighthouse CI. A build that exceeds budget fails the pipeline. Forces conscious trade-offs at PR time instead of discovering bloat in production.