Caching at scale: build and break a cache
Hands-on project: put a cache in front of a slow store with a chosen strategy, measure its hit ratio and the DB load behind it, then deliberately trigger and fix a stampede, a stale read, and a synchronized-expiry storm.
Reading that a stampede multiplies DB load and that delete-on-write races is not the same as watching your own database fall over, then fixing it. Put a real cache in front of a deliberately slow store, measure the hit ratio and the load it hides, and then break it on purpose — a stampede, a stale read, a synchronized-expiry storm — and fix each with the technique from the unit.
This project makes the whole unit operational: you’ll choose a strategy and prove the hit-ratio math against a measured database-load number, then reproduce the three signature failures — the stampede, the stale-read race, and the TTL-cliff storm — and demonstrate that the unit’s fixes actually eliminate them.
Build a cache layer in front of a slow backing store, instrument its hit ratio and the backend load behind it, and then deliberately reproduce and fix a cache stampede, a write-time stale-read race, and a synchronized-expiry (TTL-cliff) storm — closing the loop between the unit's theory and observed behavior.
- A running cache-in-front-of-store with live counters for hits, misses, and backend reads/sec, and a logged steady-state hit ratio that matches the measured backend load via backend_reads ≈ total_reads × (1 − hit_ratio).
- A reproduced stampede with a captured backend-reads spike at each hot-key expiry, and an after-fix measurement showing single-flight collapses it to ~1 recompute per expiry.
- A reproduced write-time stale-read (old value re-poisoning a key after a delete) and a fixed version using versioned keys (or a TTL backstop) where the stale value no longer persists, demonstrated by a test.
- A reproduced synchronized-expiry storm (periodic backend spike at the shared TTL boundary) and an after-jitter measurement showing the spike flattened.
- A short write-up tying each observed failure to the unit's mechanism and fix, with the before/after backend-load numbers for all three.
- Shard the cache across multiple nodes and compare modulo (hash % N) vs consistent hashing on a node add: measure the fraction of keys that miss immediately after the topology change and show modulo mass-misses while consistent hashing only disturbs ~1/N.
- Add a hot-key saturation scenario: route one extremely hot key and show a single shard saturating, then mitigate with key replication or an in-process near-cache and measure the load redistribution.
- Add a cold-start scenario: flush the cache under load and measure the backend-load spike and recovery time, then compare warming the cache first vs taking traffic cold.
- Implement probabilistic early recompute (refresh a hot key ahead of expiry) and compare it against single-flight on tail latency at the expiry boundary.
- 01How do you prove the hit-ratio/DB-load relationship empirically?
- 02How do you reproduce and fix a stampede in the load test?
- 03How do you demonstrate the stale-read race and the synchronized-expiry storm, and their fixes?
This project turns the caching unit into a repeatable build-and-break workflow. First you make the hit-ratio math real: put a cache in front of a deliberately slow store via cache-aside, instrument hits, misses, and backend reads/sec, and confirm with measured numbers that backend_reads ≈ total_reads × (1 − hit_ratio) — so a 99%→90% hit-ratio drop shows up as a 10× backend-load spike. Then you reproduce and fix the three signature failures: the stampede (a hot key’s expiry triggers a burst of simultaneous recomputes, collapsed to ~1 by single-flight), the write-time stale-read race (an old value written back after a delete, eliminated by versioned keys and a TTL backstop), and the synchronized-expiry storm (a periodic spike when uniformly-TTL’d keys expire together, flattened by jitter). Each failure is tied to the unit’s mechanism and proven fixed with before/after backend-load numbers. The stretch goals push into distribution — modulo vs consistent hashing on a rescale, hot-key saturation, and cold start. An engineer who has built this once stops treating a cache as a magic speed-up and starts treating it as a system with measurable behavior and predictable failure modes you design against up front.
Something unclear?
Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.