Load balancing
A load balancer spreads requests across a pool of servers. The choice between L4 and L7, and between round-robin and load-aware algorithms, decides your tail latency — and a naive health-check on failover can stampede the survivors into a second outage.
A team ran six identical API nodes behind a plain round-robin balancer and slept well — until one node started garbage-collecting in long pauses. Round-robin kept feeding it one request in six, perfectly evenly, straight into a server that took 4 seconds to answer instead of 40 ms. The dashboards showed five healthy nodes and one sick one, but the p99 for the whole service tracked the sick node, because a sixth of all users were routed onto it by a balancer that counted requests and never once looked at whether they came back. “Even distribution” was exactly the wrong goal: they needed load-aware distribution, and they needed the balancer to notice a slow node before their users did.
What a load balancer actually does
A load balancer sits in front of a pool of identical servers and decides, per connection or per request, which one handles it. It buys three things at once: horizontal scale (the pool grows behind a single address), fault tolerance (a dead server is taken out of rotation and traffic flows to the rest), and a stable entry point (clients talk to one virtual address, not to individual machines). It is the front door that makes the stateless-tier scaling from the previous unit actually work — you can only “add more identical nodes” if something fans traffic across them.
The two big decisions are at what layer it operates and by what rule it chooses a server. Get either wrong and the symptom is the same: load that looks balanced by request count but is wildly unbalanced by actual work, and a tail latency that no single server’s metrics explain.
L4 vs L7: where the balancer reads the request
Layer 4 (transport) balancing routes by the TCP/UDP connection: source/destination IP and port. It forwards packets (or splices a connection) without ever parsing the bytes inside. That makes it extremely fast and protocol-agnostic — it moves whatever flows over the connection, and it has no opinion about HTTP. The cost is that it cannot make decisions based on content: it can’t route /api to one pool and /images to another, can’t read a cookie, can’t terminate TLS to inspect a path.
Layer 7 (application) balancing terminates the connection, reads the HTTP request, and routes on its contents: path, host header, method, cookies. That unlocks content-based routing, per-route pools, header rewriting, TLS termination, and request-level retries. The cost is CPU and latency — it does real parsing work on every request, and it becomes a place where application concerns accumulate. (TLS termination here connects to the networking track’s TLS lesson; we stay at the composition level — the point is where the handshake ends, not how it works.)
The senior rule of thumb: L4 when you need raw throughput and protocol independence (databases, gRPC streams, anything where you don’t need to see inside); L7 when routing decisions depend on what’s in the request (web APIs, microservice gateways, anything path- or header-aware).
▸Why this works
Why does L4 cost so little and L7 so much? An L4 balancer can often hand a connection off and step out of the data path entirely (direct server return), or just shuttle packets — a few microseconds of work per connection. An L7 balancer must be on the path for the whole request: it terminates TCP and TLS, buffers and parses the HTTP message, applies routing rules, opens a separate upstream connection, and proxies the body both ways. That’s two connections, a parse, and a copy per request instead of a forward. You pay that tax for the privilege of making decisions on content — which is exactly why you push L7 features (auth, rewriting, aggregation) into it only when you actually need to see the request.
The algorithms — and why they decide your tail
Once a server is chosen, how matters more than people expect. The common algorithms, from blind to load-aware:
- Round-robin — hand each new request to the next server in turn. Simple, even by count, blind to load. Fine when every request costs the same and every server is identical; dangerous the moment one request is expensive or one node is slow (the hook).
- Weighted round-robin — give bigger servers a larger share. Handles a heterogeneous fleet, still blind to live conditions.
- Least-connections — send the next request to the server with the fewest active connections. A cheap proxy for “least busy,” and far better than round-robin when request durations vary, because a slow node accumulates open connections and naturally stops receiving new ones.
- Least-response-time / EWMA — track each server’s recent latency (often an exponentially-weighted moving average) and prefer the fastest. This is the one that would have saved the hook’s team: a node that starts taking 4 s gets less traffic, not its fair one-in-six. It’s load-aware in the truest sense — it reacts to how the server is actually behaving right now.
- Consistent hashing — hash a key (client IP, user ID, cache key) onto a ring of servers so the same key lands on the same server most of the time. Essential when the backend holds per-key state — a cache, a session, a shard — because it minimizes how many keys move when a server joins or leaves (only
1/Nof keys remap, not all of them).
Together, these algorithms form a spectrum from “count and forget” to “react in real time.” When you pick one, ask yourself: do all my requests cost the same amount of work? If not, round-robin is quietly killing your tail latency — and least-connections or EWMA is the fix.
Health checks and sticky sessions
A balancer is only as good as its picture of who’s healthy. Health checks probe each server (a TCP connect, or better, an HTTP /healthz that actually exercises the app) and eject failing nodes from rotation. The subtlety is active vs passive: active probes ask “are you alive?” on a timer; passive checks watch real traffic and eject a node that starts erroring or timing out. The best systems do both — and crucially, a health check should test the dependency chain the server needs (can it reach the DB?), not just “is the process up,” or you’ll happily route traffic to a node that returns 500s instantly.
Sticky sessions pin a given client to the same backend (via a cookie or source-IP hash) so per-user state held on that node stays reachable. The previous unit already flagged why this is a band-aid: it re-introduces a per-user single point of failure and breaks clean rebalancing. The clean alternative is the stateless tier — externalize session state to a shared store or a signed token — so any server can serve any request and you never need stickiness. Use consistent hashing for cache locality (a performance optimization you can lose safely), not for correctness (which stickiness implies and which will bite you when the node dies).
▸Common mistake
The expensive failure mode is the thundering herd on failover. One node in a pool of ten dies; the balancer correctly redistributes its traffic — but now nine nodes each absorb an extra ~11% load instantly, and if they were already near the queueing knee, that shove can tip the next one over. It fails, its load redistributes onto eight, and you get a cascading collapse where the recovery causes the outage. It’s worse with retries: every request that was on the dead node retries at once (a synchronized burst), and worse again with a cold cache on the survivors. The defenses are the ones AWS documents for retries — capacity headroom so survivors can absorb a peer’s share, jittered rather than synchronized retries, and circuit breakers / load shedding so an overwhelmed survivor sheds new work instead of dying. Plan for N−1 (or N−2) capacity, or your failover is your outage.
Order the steps to diagnose and fix a load-balanced service where p99 is high but most nodes look healthy:
- 1 Check per-node latency metrics — identify whether one node's p99 diverges sharply from the rest, revealing the slow outlier round-robin keeps feeding.
- 2 Confirm the balancing algorithm — verify it is round-robin (count-based, load-blind) rather than a load-aware algorithm like least-connections or EWMA.
- 3 Inspect the slow node for a root cause — GC pauses, CPU saturation, or a downstream dependency holding connections open.
- 4 Switch the algorithm to least-connections or least-response-time/EWMA so the balancer routes around the slow node automatically rather than giving it an even share.
- 5 Add a passive health check or latency-aware probe so a node that is slow but still accepting connections is flagged and can be ejected or deprioritised before users notice.
One node in a round-robin pool of six starts taking 4 s per request (it was 40 ms) but still accepts connections. What does the balancer do, and what's the better algorithm?
You need to route by URL path (/api to one pool, static assets to another) and read a routing cookie. Which layer must the balancer operate at, and what's the cost?
To send a given cache key or shard key to the same backend most of the time — and to remap only ~1/N of keys when a server joins or leaves — a load balancer uses _______ hashing, instead of a plain modulo that reshuffles every key on a membership change.
- 01Contrast L4 and L7 load balancing — what each can and can't do, and when to use each.
- 02Name the load-balancing algorithms from blind to load-aware, and say which protects tail latency.
- 03What is the thundering herd on failover, and how do you defend against it?
A load balancer is the front door that makes horizontal scaling real: it fans requests across a pool, ejects dead nodes, and gives clients one stable address. Two decisions dominate. Layer: L4 routes by IP/port — fast, protocol-blind, can’t see inside; L7 terminates and parses the request to route on path/header/cookie and terminate TLS, at a CPU cost. Algorithm: round-robin and weighted RR are blind to load, while least-connections and least-response-time/EWMA are load-aware and protect your tail latency by routing around a slow node instead of feeding it an even share; consistent hashing sends a key to the same backend and remaps only ~1/N of keys on a membership change, which is why cache- and shard-keyed routing depends on it. Health checks must test the real dependency chain, not just “process up”; sticky sessions are a band-aid that re-creates a per-user SPOF — externalize state instead and keep the tier stateless. And always plan for the thundering herd on failover: size for N−1, jitter retries, and shed load, or the recovery from one dead node becomes a cascading outage. Now when you see a service where p99 is mysteriously high despite “balanced” traffic, the first question is which algorithm the balancer is using — and whether it can actually see how slow one of its nodes has become.
Practice
Start at the top. Tasks go easiest → hardest: recall a fact, apply it to a case, then a senior-level stretch. Open one, attempt it, then reveal.
Something unclear?
Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.