Retry storms, circuit breakers, and load shedding

A backend has a 500 ms GC pause. One hundred clients time out. Each retries. Now the backend has 200 requests queued on an already-paused backend. The retries from the retries pile on. Within seconds, a 500 ms blip has become a full cluster outage. This is the retry storm — and automatic retries caused it.

The retry-amplification problem

A backend blips (GC pause, thread pool exhaustion, momentary overload). Requests time out. Clients and the LB retry. The retry adds load to an already-impaired backend. The backend falls further behind. More retries accumulate.

Amplification math. With N=100 clients and 1 retry each: 100 original requests + 100 retries = 200 requests on a backend that could not handle 100. With 2 retries each: 300. With K=10 microservice layers each doing 1 retry on error, worst-case amplification approaches 2^10 = 1024× the original load.

This is not a theoretical concern. It is the most common cause of sustained cluster-wide outages in microservice architectures.

Why retries make it worse, not better. Retries are correct for transient failures (momentary packet loss, brief DNS blip). They are catastrophic for overload failures: retrying into an overloaded backend adds the exact load that caused the overload. The backend cannot drain its queue because retries keep refilling it.

Circuit breakers

A circuit breaker sits at the LB or service client level and tracks failure rates for each backend. Three states:

1. Closed (normal): Requests flow through. Failure count is monitored.
2. Open: After N consecutive failures (or a failure rate threshold is exceeded), the breaker opens. All new requests receive 503 Service Unavailable immediately — no queuing, no waiting.
3. Half-open: After a cooldown period, one request is allowed through. If it succeeds, the breaker closes. If it fails, it reopens and the cooldown resets.

Why fast 503 helps. When the breaker is open, clients get 503 immediately instead of waiting for a timeout (30 s). They can route to a fallback, retry a different service, or shed the request. The overloaded backend gets no new load — it can drain its existing queue and recover.

Envoy circuit breaking config:

• max_connections: max concurrent TCP connections to a backend (e.g., 1 000). 1 001st request: 503 immediately.
• max_pending_requests: max requests queued when all connections are busy (e.g., 100). Excess: 503 immediately.
• max_requests: max concurrent HTTP/2 requests per connection (e.g., 1 000).

Retry budgets

Instead of unlimited retries, set a global retry budget: cap the total number of retries across all clients to ~10% of the total request rate.

Example: If the service handles 10 000 RPS, allow at most 1 000 RPS of retries. Any retry beyond this returns 503 immediately (fail fast). This prevents the amplification from exceeding a bounded factor regardless of how many clients are retrying simultaneously.

Exponential backoff with jitter

Clients must not retry immediately — that synchronizes all retries and creates a thundering herd. Two components:

1. Exponential backoff: Retry delay doubles on each attempt: 1 s → 2 s → 4 s → 8 s → … up to a maximum (e.g., 32 s).
2. Jitter: Add a random value random(0, base) to each delay. This desynchronizes clients so they retry at staggered times.

delay = min(base × 2^attempt + random(0, base), max_delay)

Example with base=1 s, max=32 s:

• Attempt 1: 1 s + rand(0, 1 s).
• Attempt 2: 2 s + rand(0, 2 s).
• Attempt 3: 4 s + rand(0, 4 s).

Without jitter, all 100 clients that timed out at t=30 s retry at t=31 s, creating another burst. With jitter, they retry spread across t=31–32 s.

Load shedding

The LB and backend both enforce queue depth limits. When the queue exceeds a threshold, new requests are dropped immediately with 503 rather than accepted and queued.

Without load shedding:

• Queue grows unbounded.
• Latency of every queued request increases.
• Memory bloats.
• Eventually, everything times out simultaneously.

With load shedding:

• Clients get fast 503 — they know to back off.
• The backend queue stays bounded.
• Requests that do enter the queue get served in finite time.
• System drains and recovers instead of collapsing.

cluster.api_backend.requests: 10000
cluster.api_backend.errors: 245
cluster.api_backend.retries: 189
cluster.api_backend_retry_limit: 1200
upstream_rq_retry.api_backend: 189
upstream_rq_retry_limit_exceeded: 45
upstream_rq_total.api_backend: 10000
upstream_cx: 156
health_checks.failed: 12
health_checks.success: 188
circuit_breaker.default.rq_open: 0
circuit_breaker.default.cx_open: 0
outlier_detection.ejected_count: 0
load_balancer.least_request.unbalanced_requests_delta: 234