Retry strategies: backoff, jitter, and thundering herd

A service returns 503. One thousand clients all schedule a retry for exactly 2 seconds later. At T+2s, 1000 requests hit the recovering server simultaneously — the same spike that broke it the first time. The retry mechanism just made the outage worse.

Exponential backoff: the base formula

The minimum viable retry loop waits longer before each successive attempt:

delay = min(cap, base × 2^attempt)

base is typically 100–500 ms. cap is typically 30–60 s. Attempts cap at 3–6 depending on operation criticality.

With base = 200ms and cap = 30s:

• Attempt 1 → wait 200 ms
• Attempt 2 → wait 400 ms
• Attempt 3 → wait 800 ms
• Attempt 4 → wait 1 600 ms
• …
• Attempt 8 → wait 30 s (capped)

Problem: if 1,000 clients fail at the same instant, they all pick the same delay and retry at the exact same time — the synchronized spike is the thundering herd.

Jitter: spreading retries across time

Three jitter variants (named by the AWS Architecture Blog):

Full jitter (recommended default):

delay = random(0, min(cap, base × 2^attempt))

Picks a random value from 0 to the exponential cap. Smoothest distribution. Simplest to implement.

Equal jitter:

delay = (base × 2^attempt)/2 + random(0, (base × 2^attempt)/2)

Half deterministic, half random. Harder to reason about.

Decorrelated jitter (used by AWS SDK v3):

delay = min(cap, random(base, prev_delay × 3))

Each retry’s window depends on the previous delay. Grows organically. Empirically similar load profile to full jitter.

Avoid plain exponential backoff without jitter — it is a textbook thundering-herd trap.

What to retry — and what not to

Retry conditions matter as much as retry timing:

Retry: 5xx errors, network timeouts, connection refused, 429 Too Many Requests (with Retry-After).

Do not retry: 4xx errors (except 408 and 429) — the client’s request is wrong and retrying produces the same 4xx. 410 Gone — the resource is permanently absent.

Many libraries naively retry every non-2xx, generating retry storms for permanent client bugs. Explicitly enumerate the retry condition in your configuration.

Retry-After: the server’s backoff hint

When a server returns 429 or 503, it can include Retry-After: 30 (seconds) or an absolute date. A well-behaved client honors this before applying its own backoff. gRPC uses the grpc-retry-pushback-ms trailer for the same purpose.

A client that ignores Retry-After and applies its own short backoff is the thundering-herd pattern in action — the server asked for breathing room and the client refused.

Retry budget: capping total retries across the call site

Individual retry caps (3–6 attempts) are not enough at scale. A retry budget is a token-bucket of allowed retries per second across the entire call site. When the bucket empties, the client opens a circuit and returns a deterministic error immediately — it stops hammering the downstream.

Without a retry budget, a service outage causes every client to burn through its 6-attempt budget simultaneously, amplifying traffic by 6× and turning a 30-second blip into a minutes-long outage.