Backpressure
Backpressure is the signal a slow consumer sends upstream so producers slow instead of piling work into a growing buffer. An unbounded queue is a latent disaster: it absorbs an incident, then a backlog built in minutes takes hours to drain. Bound it, shed load, flow-control.
A notification service consumed events off a queue and called a downstream API. One morning that API slowed from 50 ms to 4 seconds. The queue — unbounded, “for safety” — happily absorbed every event the producers fired while the consumer crawled. By the time the API recovered an hour later, the queue held nine hours of backlog. The system was now in a second, worse outage: every notification arriving was nine hours stale, users were getting alerts about things that no longer mattered, and the only way out was to delete the backlog and accept the data loss. The queue meant to protect availability had manufactured a longer outage. The thing they were missing was backpressure — a way for the slow consumer to tell the producers “slow down” instead of silently hoarding work.
What backpressure is
Backpressure is the mechanism by which a component that can’t keep up pushes that fact upstream, so the producer slows its rate rather than overwhelming the consumer. It’s the difference between a system that resists overload and one that silently accumulates it. In a synchronous chain, backpressure is almost automatic: if B is slow, A’s call to B blocks, A’s own callers block, and the pressure propagates back to the source naturally. The danger of asynchronous, queue-based designs is that they break this feedback loop on purpose — the producer enqueues and returns immediately, never learning that the consumer is drowning. The buffer that decouples producer from consumer (lesson 1’s whole point) also hides the consumer’s pain from the producer. Backpressure is how you put that feedback back in deliberately.
The core idea: the rate work enters a system must, over any sustained window, be ≤ the rate work leaves it. When arrival exceeds service rate, the difference has to go somewhere — and that somewhere is a queue. The only real questions are how big you let that queue grow and what you do when it’s full.
The unbounded queue: a latent disaster
An unbounded queue has no maximum size — it accepts everything. This feels safe (“we’ll never reject work!”) and is the single most dangerous default in async systems. Here is why it’s a latent disaster — fine until it isn’t:
A queue-based system has two modes. When the queue is near-empty, latency is low — the fast mode. But if arrival rate exceeds service rate for any sustained period (a slow dependency, a traffic spike, a consumer deploy), the system flips into a second mode where the backlog grows and end-to-end latency climbs without bound. This bimodal behaviour is the trap: nothing looks wrong until you cross the line, and then the metric that matters (latency to process a message) degrades catastrophically while the producer, blissfully unaware, keeps enqueueing.
The brutal arithmetic of recovery: if a system falls an hour behind, it needs roughly double its normal capacity for an hour just to catch up — and if a spike queued 10× the consumer’s capacity, draining it takes 10× as long as the spike lasted. A short incident becomes a long one. A 30-minute pile-up can mean a 5-hour recovery. Worse, by the time the backlog is huge, the data may be stale enough to be useless — you process messages too late for the result to matter, which is exactly the availability hit the queue was supposed to prevent. The unbounded queue didn’t absorb the incident; it laundered a brief failure into a prolonged one.
▸Why this works
Why is an unbounded buffer worse than rejecting work? Because rejecting work is honest and immediate — the producer learns instantly that the system is at capacity and can back off, retry later, or shed. An unbounded queue is dishonest: it accepts the work, returns success, and silently defers the failure to a future moment when the backlog is so deep that recovery is painful and the queued work may already be worthless. Synchronous systems mostly self-heal from overload because they drop excess load at the door and recover fast; asynchronous systems accumulate it and recover slowly. Memory makes it concrete — an in-process unbounded queue under sustained overload doesn’t just slow down, it grows until the process runs out of memory and crashes, turning a slowdown into a hard outage. The bound is not a limitation; it’s the thing that keeps the failure small and recoverable.
The fixes: bound it, shed load, flow-control, rate-limit
A handful of mechanisms restore the feedback loop. They compose; mature systems use several.
- Bounded queues. Give every queue a maximum depth. When it’s full, the enqueue fails fast — and that failure is the backpressure: the producer now knows and can react. A bound converts “silent infinite backlog” into “explicit, immediate signal.” Choosing the bound is choosing how much burst you absorb before you push back.
- Load shedding. When you’re at capacity, deliberately reject (shed) some work rather than degrade everything. Reject the lowest-value work first; protect the critical path. Crucially, shedding load cheaply at the front door is far better than letting expensive half-done work pile up — fail fast and clearly. (See the dedicated availability work on load shedding for the full treatment.)
- Rate limiting at the consumer / fairness. In a multi-tenant system, one tenant’s spike can build a backlog that starves everyone. Per-tenant rate limits and isolating workloads (separate queues, or hashing tenants across a small pool of queues so a hot tenant only contaminates a few) keep one noisy neighbour from manufacturing a system-wide backlog.
- Flow control (log-based pull). Pull-based consumers (Kafka) get backpressure almost for free: the consumer asks for the next batch when it’s ready, so it can never be pushed faster than it can process. The “queue” is the retained log, which doesn’t grow in memory — but the offset lag does, and consumer lag is the metric you alarm on: it’s the direct measure of how far behind you’ve fallen and whether you’re draining or drowning.
Together these mechanisms close the loop that async decoupling broke: the system can feel its own pain again and act before the backlog becomes unrecoverable. Without at least a bound and a shed strategy, every other resilience investment — retries, failover, redundancy — can be neutralised by a single slow dependency accumulating minutes of work.
The mindset shift
The deep lesson is that a queue is not infinite headroom. Juniors add a queue and feel safe; seniors add a queue and immediately ask: what’s its bound, what happens when it’s full, who feels the backpressure, and what’s the recovery time if it fills during an incident? The queue’s job is to absorb brief mismatches between arrival and service rate — a burst, a momentary slow dependency. It is not a place to store a sustained overload, because a sustained overload means arrival > service forever, and no buffer survives forever. When the rates won’t reconcile, your only honest options are to add capacity (drain faster) or shed load (accept less). The buffer just buys you the minutes to choose.
▸Common mistake
The retry-storm amplifier, the classic way a brownout becomes an outage. When the consumer slows, clients time out and retry — which adds load precisely when the system is already overloaded, accelerating the queue’s collapse. Each retry is a new arrival on top of the original, so naive retries multiply offered load at the worst moment. The fixes are exponential backoff with jitter (spread retries out, don’t thunder), a retry budget (cap retries as a fraction of traffic, not per-request), and circuit breakers (stop calling a failing dependency entirely for a cooldown). Adding retries without these is how teams turn a recoverable slowdown into a self-sustaining overload that won’t drain until they shut traffic off.
Your async pipeline uses an unbounded queue 'so we never drop events.' A downstream slows for 30 minutes during peak. What's the most likely outcome, and the root fix?
A consumer slows, clients time out and retry, and the system collapses faster than the original slowdown explains. What's happening and which fixes address it?
An unbounded queue can never push back because it is never full; the fix is to make every queue _______ so that a full queue fails fast — and that fast failure IS the backpressure signal that tells producers to slow down or shed load.
This is composition altitude: where backpressure belongs in a design and why the bound matters. Load shedding has its own depth in the availability track, and broker-specific flow-control (Kafka consumer lag tuning, RabbitMQ flow control) lives in the dedicated queues track — reach there to operate a specific broker; stay here to decide that your design needs a bound and a shed strategy at all. Now when you see a queue described as “unbounded, for safety,” treat that as the risk it is — and ask: what’s the recovery time if this fills during peak?
- 01What is backpressure, and why do async/queue designs lose it?
- 02Why is an unbounded queue a latent disaster?
- 03List the mechanisms that restore the feedback loop.
Backpressure is the signal a component that can’t keep up sends upstream so producers slow down rather than overwhelming it. Synchronous chains get it almost for free (a slow call blocks back to the source); async/queue designs break the feedback loop on purpose — the decoupling buffer that hides the consumer’s slowness from the producer is exactly what removes the natural signal. The invariant to hold: over any sustained window, arrival rate ≤ service rate, or the surplus piles into a queue. An unbounded queue is the most dangerous default — a latent disaster with bimodal behaviour: fine in fast mode, then once arrival exceeds service the backlog grows without bound, recovery needs roughly double capacity for as long as you were behind (a 10× spike → 10× drain), the data may go stale, and an in-memory queue can OOM — so it launders a brief failure into a long outage. The fixes restore the loop: bound every queue (full = fail fast = the signal), shed low-value load cheaply at the front door, rate-limit and isolate per tenant so one noisy neighbour can’t starve everyone, and use pull-based flow control while alarming on consumer lag. Guard against the retry storm (backoff + jitter, retry budget, circuit breakers) that turns a brownout into a self-sustaining outage. The mindset: a queue is not infinite headroom — it absorbs brief bursts, never sustained overload; when the rates won’t reconcile, your honest options are add capacity or shed load, and the buffer just buys the minutes to choose. Now when you see any queue described as “unbounded, for safety,” ask the one question that matters: what is the recovery time if this fills during an incident — and is the answer acceptable?
Practice
Start at the top. Tasks go easiest → hardest: recall a fact, apply it to a case, then a senior-level stretch. Open one, attempt it, then reveal.
Something unclear?
Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.