Distributed caches
One cache node is not enough memory and is a single point of failure. Sharding (via consistent hashing) spreads keys across nodes; replication survives node death. But scaling out the cache invites cache stampedes, hot keys, and cold starts — failures that only appear at scale.
A popular product page was cached with a single hot key and a 60-second TTL. At a traffic peak of 50,000 reads per second the key expired, and in the few hundred milliseconds it took to regenerate, every one of those 50,000 requests missed the cache, found nothing, and slammed the database with the same expensive query — all at once. The database, sized to handle the trickle of misses that normally leak through a 99%-hit cache, fell over. The page came back, the key warmed, traffic resumed… and 60 seconds later it happened again. Nobody had written a bug. The cache was working perfectly. The failure was structural: at scale, one expiring key can become a synchronized weapon pointed at your database.
Why one cache node isn’t enough
By the end of this lesson, when someone proposes “just add another cache node,” you’ll know exactly which questions to ask — and which silent failure modes they may not have considered.
A single cache node hits two walls the moment you scale. Capacity: the working set outgrows one machine’s RAM, so you can’t hold it all. Availability: that one node is a single point of failure — when it dies or restarts, your hit ratio drops to zero and (per the previous lesson’s arithmetic) the full read load lands on the database. A distributed cache solves both with two orthogonal tools: sharding for capacity and replication for survival.
Sharding partitions the keyspace across N nodes — each key lives on exactly one shard, so total capacity is N × one-node capacity. Replication keeps copies of a shard on more than one node, so losing a node loses no data and reads can fan out across replicas. The two compose: a production cache is usually sharded and each shard replicated. The hard question sharding raises is which key goes to which node — and naive answers fail badly.
Sharding: why consistent hashing, not modulo
The obvious sharding rule is node = hash(key) % N. It works until N changes. Add or remove one node and N changes, so hash(key) % N changes for almost every key — a near-total remapping that invalidates the whole cache at once and stampedes the database. Consistent hashing (covered in the data-distribution unit) fixes this: keys and nodes are placed on a hash ring, a key belongs to the next node clockwise, and adding or removing a node only remaps the keys in one arc — roughly 1/N of the keyspace — instead of all of them. That property is exactly what a cache needs: scaling the cluster should disturb a slice of the cache, not nuke it.
▸Why this works
Why is “only 1/N keys move” the property that matters most for a cache specifically? Because in a durable datastore a remap means copying data (expensive but correct); in a cache a remap means a miss storm. Every key that moves to a new (empty) node is now a guaranteed miss until it’s reloaded from the database. With modulo hashing, scaling from 4 to 5 nodes remaps ~80% of keys, so ~80% of your cache instantly misses and that load hits the database in a thundering wave — the exact failure you added nodes to avoid. Consistent hashing caps the damage at the moved arc (~1/N), and virtual nodes (many ring positions per physical node) smooth the load so one node doesn’t inherit a disproportionate arc. Sharding a cache without consistent hashing turns a routine scale-up into an outage.
The cache stampede (thundering herd), and how to stop it
The hook is a cache stampede (also thundering herd or dogpile): a hot key expires, and N concurrent readers all miss simultaneously and all recompute the same value at once, multiplying load on the backend by N. It is a scale-only failure — invisible at 10 RPS, lethal at 50,000. Three standard fixes, in increasing sophistication:
- Request coalescing (single-flight) — when many requests miss the same key at the same time, let one of them recompute while the others wait for that single result instead of each launching their own DB query. N concurrent misses become one DB call. This is the workhorse fix.
- Early/probabilistic recompute — don’t wait for the key to fully expire. As it nears expiry, let a single request refresh it ahead of time in the background, so the hot value never actually goes absent. Probabilistic early expiration (each reader rolls a dice that gets likelier as expiry approaches) means exactly one tends to refresh, with no coordination.
- Locks / leases — the recomputing request takes a short lock (or a lease) on the key; others see the lock and serve the slightly-stale old value, or wait briefly, rather than piling onto the backend.
All three approaches share one idea: turn N simultaneous database calls into one. Without that reduction, every popular key with a TTL is a ticking bomb. When you’re reviewing a system design, ask whether hot keys are protected — not just whether a cache exists.
These mechanics are the subject of the dedicated caching track’s stampede lessons (single-flight, probabilistic early recompute, stale-while-revalidate); at the system-design altitude the point is to recognize the stampede as a structural risk of any hot, TTL’d key at scale and to know that coalescing the misses is the answer.
Hot keys, cold starts, and client-side vs server-side
Sharding assumes load spreads evenly across keys — but real traffic has hot keys: one celebrity’s profile, one viral product, one global config value that every request reads. A hot key concentrates all its traffic on the single shard that owns it, so that one node saturates while the rest idle — sharding doesn’t help because the key can’t be split. Fixes: replicate the hot key to multiple nodes and read from a random replica, or also cache it locally in each app process (a small client-side/near cache) so most reads never reach the shared cache at all. That local tier is the client-side vs server-side distinction: a server-side cache (shared Redis cluster) is consistent across all app nodes but one network hop away; a client-side/in-process cache is a hop closer and absorbs hot keys, but each app node has its own copy and so they can disagree — you’re back to per-node staleness.
Cold start is the other scale failure: a freshly deployed or restarted cache is empty, so for the first minutes every read misses and the database eats the full unmitigated load — the same risk as a stampede but cluster-wide. Mitigations: warm the cache before taking traffic, roll restarts gradually (not all nodes at once), and lean on consistent hashing so a single replaced node only cold-starts its own arc.
▸Common mistake
A subtle and common mistake is assuming replication makes your cache strongly consistent. Replicas in a cache are almost always asynchronous: a write to the primary shard propagates to replicas with a lag, so a read served from a replica can return a slightly older value than the primary. For a cache this is usually fine — it’s a cache, staleness is already on the table — but teams that read-from-replica for scale and then assume “the cache is the source of truth” get bitten when a just-written value isn’t on the replica yet. The cache is never the source of truth (that’s the next lesson’s anti-pattern); replication buys you survival and read throughput, not a consistency guarantee. Size replica lag into your freshness budget, don’t pretend it’s zero.
Your cache shards keys with hash(key) % N. You add one node to go from 4 to 5 nodes during a traffic peak, and the database immediately falls over. Why, and what should you have used?
A single hot key with a 30s TTL serves 40,000 RPS. Every 30 seconds the database briefly spikes to thousands of identical queries. What is this, and what's the most direct fix?
To shard a cache without nuking it on every scale change, you route keys with _______ hashing, which moves only ~1/N of keys (one arc of the ring) when a node is added or removed — instead of modulo's near-total remap that would stampede the database.
- 01Why shard with consistent hashing instead of modulo, and what does replication add?
- 02What is a cache stampede and what are the three fixes?
- 03Explain hot keys, cold start, and client-side vs server-side caching.
One cache node hits a capacity wall (working set outgrows RAM) and is a single point of failure, so a distributed cache uses two orthogonal tools: sharding for capacity and replication for survival. Shard with consistent hashing, not modulo — because for a cache a remap is a miss storm, and modulo remaps ~80% of keys on a 4→5 scale-up while consistent hashing moves only the ~1/N keys in one ring arc (virtual nodes even out the load). Replication keeps each shard’s copies for node-loss survival and read fan-out, but it’s typically asynchronous, so replicas lag — the cache is never the source of truth. Scaling out invites failures invisible at small scale: the cache stampede (a hot TTL’d key expires and N concurrent reads recompute it at once, multiplying DB load by N) is stopped by request coalescing / single-flight, probabilistic early recompute, or locks; hot keys saturate their owning shard and want replication or an in-process near-cache; and cold start leaves a fresh cache empty, mitigated by warming, gradual restarts, and consistent hashing’s per-arc scope. The deeper mechanics live in the dedicated caching track; the genuinely hard remaining problem — keeping all these copies honest — is the next lesson, invalidation. Now when you see a cache cluster being scaled up with a routine node add, you’ll know to ask: what hashing scheme is in place, and what fraction of keys will cold-miss the moment that node joins?
Practice
Start at the top. Tasks go easiest → hardest: recall a fact, apply it to a case, then a senior-level stretch. Open one, attempt it, then reveal.
Something unclear?
Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.