Timeouts and tail latency: budgets, deadlines, and the fan-out trap

Each service in the chain is healthy: every dependency answers in 10 ms at the 99th percentile. The product page fans out to 100 of them in parallel and waits for all. Yet 63% of page loads take over a second. No single service is slow. The math of the tail is slow — and most engineers’ intuition about averages hides it completely.

A timeout on every hop, or a hang on every incident

A network call without a timeout is a bug waiting for an incident. When a dependency stops responding (not refuses — hangs), a call with no timeout waits forever, holding a thread or connection. Enough such calls exhaust the pool, and a single slow dependency takes down a healthy service. Every outbound call — DB query, cache, HTTP, RPC — needs an explicit timeout. The default in most clients is no timeout, which is the worst default in backend engineering.

Timeouts must compose into a budget

Per-hop timeouts set in isolation lie. If a request has a 1 s SLA but calls service A (timeout 1 s) which calls service B (timeout 1 s), then when B is slow, A waits the full second and the client has already given up — A is now doing work nobody is waiting for. The fix is a timeout budget (a deadline): the entry point allocates a total, and each hop passes the remaining time downward. gRPC formalizes this as a deadline propagated in metadata; every service computes its local timeout as min(local default, remaining budget).

Why the tail, not the average, is the SLA

Users do not experience your average. They experience their own request, and the slow ones are what they remember and what trips alerts. So latency is reported as percentiles: p50 (median), p99 (1 in 100 is worse), p99.9. The gap between p50 and p99 is “tail latency,” caused by GC pauses, queueing, cache misses, lock contention, and retries.

The danger is tail amplification under fan-out. If one request to a service is slow with probability p, then a request that fans out to N services in parallel and waits for all is slow if any one is slow — probability 1 − (1 − p)^N. With a per-service p99 (p = 1%) and N = 100, that is 1 − 0.99^100 ≈ 63%. This is the Hook’s number, straight from Dean and Barroso’s The Tail at Scale: a service fanning out to 2,000 leaves about 20% of requests over a second even when each backend’s p99 is fine.

Defending the tail: hedging, not just timeouts

A timeout caps the worst case but does not improve the typical tail. The technique from The Tail at Scale is hedged requests: send the request, and if no answer arrives by the p95 latency, send a second copy to another replica and take whichever returns first. Because only the slow ~5% get hedged, the extra load is small (~5%) while the tail collapses — in Google’s measurements, sending a hedge after a 10 ms delay cut p99.9 from 1,800 ms to 74 ms at the cost of ~2% more requests. Tied requests go further: the duplicates tell each other to cancel once one starts executing, trimming wasted work.