Traffic & edge: a load balancer and edge plan
Hands-on project: stand up a real load balancer in front of a small pool, prove that a load-aware algorithm beats round-robin under an uneven node, then design and measure a CDN cache policy — hit ratio, TTL, and what must never hit the shared edge.
Reading that a load-aware algorithm protects your tail and that a CDN’s hit ratio is non-linear is not the same as proving it on a running system. Stand up a real load balancer in front of a small pool, inject one slow node, and watch round-robin versus least-response-time diverge — then design a cache policy for a mixed app and measure the hit ratio you actually get.
This project makes the whole unit operational: you’ll configure a real balancer, demonstrate the tail-latency difference between a blind and a load-aware algorithm, verify that a meaningful health check ejects a broken node, and then design a per-response cache policy and measure its hit ratio — including proving what must never reach the shared edge.
Build a small but real traffic edge: a load balancer (nginx/HAProxy/Envoy or a cloud LB) fronting a pool of identical app instances, demonstrate empirically that a load-aware algorithm protects tail latency where round-robin does not, verify health-check ejection, and then design and measure a CDN-style cache policy for a mixed app — closing the loop between the unit's claims and observed behavior.
- A running balancer + pool reachable through one address, with the algorithm and health-check configuration shown.
- Two load-test results (round-robin vs load-aware) under the same injected slow node, with p50/p99 and throughput, visibly showing round-robin's p99 tracking the slow node and the load-aware algorithm holding the tail near the healthy nodes.
- A demonstration that breaking a node's probed dependency causes the balancer to eject it (logs or metrics), with traffic continuing on the remaining nodes and recovering when the node is healed.
- A one-page cache-policy table covering all five content types with explicit Cache-Control/TTL/Vary, and a one-line justification per row, including which responses are never cacheable on a shared edge and why (leak / correctness / write).
- A measured before/after cache hit ratio for the public content with the change you made, plus the implied origin-RPS reduction (showing you understand the non-linear miss-fraction economics).
- A short failover write-up: the N−1 headroom calculation and the recommended pool size, tying back to the thundering-herd risk.
- Add consistent hashing: put a keyed cache tier behind the balancer, show that hashing on the key gives a high hit ratio while least-connections collapses it, and measure how many keys remap when you add/remove a node (~1/N).
- Add a thundering-herd scenario: expire a hot object's TTL across many edge/proxy caches at once, observe the synchronized origin spike, then enable request coalescing (and/or an origin shield) and show the origin sees one fetch instead of many.
- Make the gateway redundant: run two balancer/gateway instances behind a higher-level balancer, kill one mid-load, and show no full outage — contrasting it with the single-instance SPOF.
- Add jittered retries to the load generator and show how synchronized (un-jittered) retries during a node failure accelerate the cascade, while jitter spreads the recovery load and avoids the brownout-to-outage feedback loop.
- 01How do you empirically show a load-aware algorithm protects tail latency?
- 02What does a sound CDN cache-policy design specify per response type?
- 03Why measure the hit ratio and compute N−1 headroom rather than trust the defaults?
This project turns the traffic-and-edge unit into a repeatable workflow: stand up a load balancer over an identical pool, then prove the unit’s central claim by injecting one slow node and showing round-robin pins the service p99 to it while a load-aware algorithm (least-connections/least-response-time) routes around it. Verify that a meaningful health check — one that probes a real dependency, not a static 200 — ejects a broken node and that traffic continues. Then design a per-response CDN cache policy for a mixed app (fingerprinted assets, public page, per-user page, must-be-correct value, write endpoint), measure the hit ratio before and after a TTL/cache-key improvement, and translate it into the non-linear origin-load reduction. Finally, do the N−1 failover capacity math so the survivors can absorb a lost node without a thundering-herd cascade. Doing the configuration and measurement, not just reading the claims, is the whole discipline — an engineer who has built this once chooses layers, algorithms, gateways, and cache policies deliberately instead of discovering their failure modes during an incident.
Something unclear?
Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.