Retries and resilience: jittered backoff, retry budgets, and circuit breakers

Retry safety is a property of the operation

Before you write a retry loop, ask one question: if this operation runs twice, is the outcome the same as running it once? The answer determines whether you are building resilience or a double-charge bug. A timeout is not a failure report — it is the absence of a report. The request may have died on the wire, or it may have succeeded after you stopped listening. Retrying is therefore only safe when running the operation twice is equivalent to running it once. GET, PUT and DELETE promise that by HTTP contract (if the handlers honor it); POST promises nothing — retrying a timed-out POST /charge because “it failed” is how customers get charged twice. The industry fix is the idempotency key: the client generates a unique key per logical operation, sends it with every attempt, and the server stores the result under that key, replaying the stored response for duplicates. Exactly-once delivery is a lie the network does not permit; at-least-once plus server-side deduplication is what idempotency keys actually implement. Classify every call site before writing any retry loop: idempotent by nature, idempotent via key, or not retryable at all.

Backoff with full jitter

Failures synchronize clients: a downstream restart, a cache flush, a deploy — and a thousand callers fail in the same hundred milliseconds. If they all retry on the same schedule, they return as a wave, and exponential backoff alone only spaces the waves: spikes at one second, two, four, each one a thundering herd against a recovering service. Jitter is what breaks the synchronization. The AWS Architecture Blog analysis that named these strategies found full jitter — sleeping a uniformly random time between zero and the exponential ceiling — close to optimal: clients spread evenly across the window, the herd becomes a drizzle, and total work done by the system drops dramatically compared to unjittered backoff.

const (
	baseDelay = 100 * time.Millisecond
	maxDelay  = 5 * time.Second
)

// Full jitter: uniform in [0, min(maxDelay, base*2^attempt)).
func backoff(attempt int) time.Duration {
	ceiling := baseDelay << attempt // 100ms, 200ms, 400ms, ...
	if ceiling > maxDelay {
		ceiling = maxDelay
	}
	return rand.N(ceiling) // math/rand/v2: uniform in [0, ceiling)
}

Retry budgets: the amplification cap

Jitter fixes synchronization; it does nothing about amplification. That distinction matters — here is the mechanism that killed the catalog service. With three attempts per request, a downstream at 100% failure receives three times its normal traffic — precisely when its capacity is lowest. Stack retries across layers and they multiply: three attempts at the client and three at the gateway is up to nine arrivals downstream for one user action. The SRE-book discipline is two caps. Per request: at most three attempts, then fail upward. Per client: a retry budget — retries may not exceed roughly 10% of requests in a sliding window; when the budget is spent, failures return immediately without retrying. Under total downstream failure, a budgeted client sends at most 1.1x its normal load instead of 3x, and retries help with what they actually help with — transient blips — while staying harmless during real outages. And retry at one layer only, the one that owns the user-visible deadline; everything below should fail fast upward.

Timeouts: per attempt, under one deadline

Retries and timeouts are one design problem, and the context tree from the first lesson is its skeleton. The caller-facing SLO becomes the overall deadline on ctx; each attempt derives a shorter per-attempt context, so a hung attempt dies quickly and the retry can land on a healthier replica; and the loop checks the parent between attempts so it never sleeps past the deadline:

func callWithRetry(ctx context.Context, do func(context.Context) error) error {
	var lastErr error
	for attempt := 0; attempt < 3; attempt++ {
		attemptCtx, cancel := context.WithTimeout(ctx, 500*time.Millisecond)
		lastErr = do(attemptCtx)
		cancel()
		if lastErr == nil || !retryable(lastErr) {
			return lastErr
		}
		select {
		case <-time.After(backoff(attempt)):
		case <-ctx.Done():
			return errors.Join(ctx.Err(), lastErr) // deadline beats persistence
		}
	}
	return lastErr
}

The proportions matter: an overall 2-second deadline with 500 ms attempts leaves room for three attempts plus backoff; a 1-second deadline with 900 ms attempts is a retry loop in name only. Budget the deadline like money — attempts, backoff, and a reserve for the caller’s own work.

Circuit breakers: when silence beats persistence

A retry budget still sends its baseline share into a dependency that is fully down — and each of those calls burns a full timeout before failing. A circuit breaker adds the missing state machine. Closed: traffic flows, failures are counted in a window. Open: the failure threshold tripped; calls fail in microseconds without touching the network, and the downstream gets the one thing that actually helps it recover — silence. Half-open: after a cooldown, a few trial requests probe; success closes the breaker, failure reopens it. A breaker beats a budget when the dependency is binary-dead and when your own latency matters — failing fast in microseconds instead of burning 500 ms per doomed call keeps your goroutines, connection pools and callers alive. The honest costs: state per dependency, two thresholds and a cooldown to tune, false trips on low-traffic paths where three failures out of five requests is noise, and the obligation to decide what the fallback actually is when the breaker is open.