The service under overload: load shedding and graceful degradation

There is a load level above which your service cannot serve every request, no matter how it is written — and the only choice you have is how it fails when it gets there. The naive instinct is to try to serve everyone: accept every connection, queue every request, retry every failure. That instinct is exactly what produces the collapse from two lessons ago, because accepting work you cannot finish does not help the user whose request you accepted and steals the resources of the users whose requests you could have finished. The counterintuitive truth of overload is that rejecting some requests is how you serve the rest — a service that sheds 20% of load at the door and serves the other 80% fast is vastly healthier than one that accepts 100% and serves all of it too slowly to be useful, then falls over and serves 0%. This is the shift from thinking about throughput (requests handled) to goodput (requests handled usefully, within their deadline). And here is the synthesis: every mechanism you learned is secretly a load-control knob. The pool bounds concurrent work. Timeouts free resources from doomed requests. The breaker sheds load from a failing dependency. Shutdown sheds load from a departing instance. Overload is the condition that makes all of them act at once, and operating a backend means tuning them so the service bends instead of breaking.

Throughput is the wrong target; goodput is the right one

The metric that matters under overload is not how many requests you accept but how many you complete usefully — fast enough that the answer still matters to the caller. Call that goodput. A request that you accept, hold for 8 seconds, and finally answer after the client has already timed out and retried is pure waste: it consumed a connection, a loop slot, and CPU, and produced negative value, because it also delayed real work. Beyond the saturation point, raising accepted load lowers goodput — the classic overload curve where throughput climbs, peaks, then falls off a cliff as the system spends itself on work that no longer matters. The entire discipline of overload handling is keeping the system at the top of that curve instead of sliding down the far side, and that means refusing work you cannot complete in time.

Load shedding: reject early, reject cheaply

Load shedding is the deliberate, early rejection of excess requests so the accepted ones succeed. Three properties make it work:

• Early. Shed at the edge, before the request consumes a pooled connection or a downstream call. A request rejected at admission costs almost nothing; a request rejected after it has acquired resources has already done the damage. This is why an admission check or concurrency limiter sits in front of the expensive layers.
• Cheap and honest. The rejection is a fast 503 Service Unavailable with a Retry-After, not a slow error. It tells the client don’t hammer me — which is the opposite of the silent slowness that triggers retry storms.
• Prioritized. Not all load is equal. Shed the least valuable first — background refreshes before user-facing reads, anonymous before paid, retries before first attempts. A good shedder drops the right 20%, preserving goodput where it counts.

Backpressure: pushing the limit upstream

The dual of shedding is backpressure: instead of accepting work and dropping it, you signal upstream to slow down. A bounded queue that refuses new entries when full, a pool that makes acquisition fail fast instead of queueing unboundedly, an HTTP layer that stops reading the socket — each propagates “I am full” back toward the source, so the pressure is felt where it can actually be reduced (the client backs off, the upstream service routes elsewhere) rather than absorbed silently until something breaks. Shedding and backpressure are two halves of the same idea: make the limit explicit and enforce it at the boundary, rather than letting an implicit limit (memory, file descriptors, the pool) enforce itself by collapsing.

Every mechanism is a load-control knob

The synthesis the whole unit has been building toward: the seven mechanisms are not just correctness tools, they are the actuators of load control, and overload is when they combine:

• Pool / concurrency limit — the hard cap on work in flight. The single most important overload defense: it converts “unbounded slowdown” into “bounded work plus explicit rejection.”
• Timeouts — free resources from requests that will not finish in time, so doomed work stops stealing capacity from viable work.
• Circuit breaker — sheds load from a failing dependency, which is overload localized to one downstream.
• Retries (bounded, with backoff and jitter) — the load amplifier you must cap, because uncapped retries turn overload into collapse.
• Graceful shutdown — sheds load from a departing instance without dropping its in-flight work onto the floor.
• Observability — the feedback that tells you where you are on the goodput curve, so shedding can be triggered by real saturation, not guesses.

Tuning these together — pool sized to the slowest dependency, timeouts nested in a budget, retries capped, a shedder at the edge keyed off saturation metrics — is what makes a service degrade gracefully: serve less, but serve it well, and never fall to zero.