Latency vs throughput
Latency is time per request; throughput is requests per second. They are not opposites — Little's Law binds them through concurrency, and queueing theory explains why latency explodes long before throughput maxes out. Confuse them and you optimize the wrong number.
A team saw their API’s average response time was a healthy 40 ms, so when traffic doubled they were blindsided: the average stayed at 40 ms, but support tickets exploded. The p99 had gone from 180 ms to 2.3 seconds. They had been watching the mean — the one number that hides exactly the customers who are suffering. The fix wasn’t “make it faster”; it was understanding that latency and throughput are different axes, and that the system was sliding up the steep part of a curve they didn’t know existed.
Two different axes
Before you can fix what the team got wrong, you need to know which number you’re actually optimizing — and why confusing them sends you in the wrong direction.
Latency is how long one operation takes: send a request, get a response, measure the gap. Units are time — milliseconds, microseconds. Throughput is how many operations complete per unit time — requests per second (RPS), queries per second (QPS), bytes per second.
They feel like the same thing (“fast”), but they are orthogonal. A delivery truck carrying 10,000 hard drives across the country has enormous throughput (terabytes moved) and terrible latency (two days for the first byte). A fibre link has tiny latency and, for one small request, modest throughput. AWS frames it the same way: throughput is the volume of data moved, latency is the delay before it starts arriving — and you tune them with different levers.
When you see a service performing well on throughput but still getting complaints, check whether you’re looking at the mean latency or the percentiles — that’s usually where the gap hides. The junior mental model — “higher throughput means lower latency” — is wrong often enough to be dangerous. Adding a batch step raises throughput and raises latency. Adding more replicas raises throughput and may leave single-request latency untouched. To reason about a system you must track both numbers, separately.
Little’s Law: the bridge
The two axes are not independent — they are tied together by the concurrency in flight. Little’s Law states, for any stable system:
L = λ × W
L = average number of requests in the system (concurrency)
λ = throughput (arrival rate = completion rate when stable)
W = average latency (time each request spends in the system)It is almost embarrassingly general — no assumptions about distribution, scheduling, or arrival pattern. Rearranged, λ = L / W: throughput equals concurrency divided by latency. This single equation kills a lot of bad intuition. If each request takes W = 200 ms and you can hold L = 100 requests in flight, your ceiling is λ = 100 / 0.2 = 500 RPS — no amount of “optimization” beats that without changing L or W. Want more throughput? Either make each request faster (lower W) or run more concurrently (raise L, via more threads, connections, or machines).
▸Why this works
Why does Little’s Law hold with no assumptions? Because it is an accounting identity, not a physical law. Over a long window, every request that enters must leave (stability). The area under the “requests in system over time” curve can be summed two ways — by requests (count × time each stayed) or by time (integral of concurrency) — and the two sums must be equal. That equality is L = λW. This is why it applies to a CPU, a thread pool, a database connection pool, a Kafka partition, or a checkout line identically — and why a connection pool of size L with per-call latency W has a hard throughput ceiling you can compute on a napkin.
Why latency explodes before throughput maxes out
Here is the part that surprises people: as you push throughput toward a resource’s limit, latency does not rise linearly — it goes vertical. Queueing theory (the M/M/1 model) gives the shape:
W = 1 / (μ − λ)
μ = service rate (max throughput of the resource)
λ = arrival rate (offered load)At 50% utilization (λ = μ/2), waiting time is small. At 90%, the denominator is 0.1μ — latency is 10× the unloaded service time. At 99%, it’s 100×. The knee of this curve is the single most important shape in capacity planning: a server that looks “half idle” at 50% CPU has very different tail behavior from one at 85%, and the gap between them is where outages live. This is why mature systems target 60–70% utilization with headroom rather than squeezing to 95% — the last slice of throughput costs unbounded latency.
Averages lie; percentiles tell the truth
The hook’s team watched the mean. The mean is the worst summary statistic for latency, because latency distributions are heavily right-skewed: most requests are fast, a few are catastrophically slow, and those few are exactly the ones a user retries (adding load) or abandons (losing money). Report percentiles instead: p50 (median, the typical experience), p99 (1 in 100 — your power users hit this several times per session), p999 (1 in 1000).
Tail latency is not a curiosity; it compounds. If a single page makes 100 backend calls and waits for all of them, the page is as slow as the slowest of 100 calls — so the page’s median is governed by each backend’s p99. This is tail-at-scale amplification, and it is why p99 of your dependencies is your p50.
▸Common mistake
A measurement trap that silently lies: coordinated omission. A load generator that sends a request, waits for the response, then sends the next one cannot record the latency of requests it never sent during a stall. When the server freezes for 1 second, an open-loop world saw thousands of requests each suffering up to 1 s; the closed-loop generator saw one slow request. Your p99 looks fine while users are screaming. Tools like wrk2 and proper open-loop load testing exist specifically to avoid this — if your benchmark holds throughput constant regardless of server speed, distrust its tail numbers.
A service holds at most 200 concurrent requests (connection-pool limit). Measured average latency is 50 ms per request. By Little's Law, what is its maximum sustainable throughput?
Your p50 latency is flat as load rises, but p99 has tripled while CPU sits at 88%. What is the most likely cause?
Little's Law says L = λ × W, where L is concurrency, W is latency, and λ is _______ — so for a fixed concurrency limit, the only way to raise this quantity is to lower the latency of each request.
- 01State Little's Law and what each term means.
- 02Why does latency rise non-linearly as you approach max throughput?
- 03Why report percentiles instead of the average latency?
Latency and throughput are different axes: latency is time per request (measured in ms/µs), throughput is requests completed per unit time (RPS/QPS). They are joined by Little’s Law, L = λW, an accounting identity that means λ = L / W — for a fixed concurrency limit (a pool size), the only way to more throughput is lower per-request latency or more capacity. Queueing theory explains the danger: W = 1/(μ−λ), so latency stays calm until you near a resource’s limit, then goes vertical — which is why mature systems run at 60–70% utilization, not 95%. Finally, measure percentiles, not the mean: latency is right-skewed, the slow tail is what users feel, and under fan-out each dependency’s p99 becomes your p50. Now when you see a healthy-looking average latency while support tickets climb, you’ll know exactly where to look: pull up p99 and check the utilization curve before declaring anything fixed.
Practice
Start at the top. Tasks go easiest → hardest: recall a fact, apply it to a case, then a senior-level stretch. Open one, attempt it, then reveal.
Something unclear?
Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.
Apply this
Put this lesson to work on a real build.