Eviction policies and TTL
A cache is finite, so it must throw things away. Eviction policy (LRU, LFU, FIFO, random) decides what leaves when memory fills; TTL decides when a value goes stale. Both exist to keep the working set hot — and your hit ratio, not your cache size, is the number that matters.
An engineer set a cache to evict the oldest entry first (FIFO) and went home happy. The cache held the homepage, which is read a million times an hour, plus a nightly report nobody touches until 9 a.m. Because the homepage was loaded at boot — it was the oldest entry — FIFO threw it out the moment the cache filled, then reloaded it on the next read, then evicted it again. The cache was busy churning out its single most valuable item while faithfully retaining the report. The fix was one word in a config — change the eviction policy from FIFO to LRU — but it exposed the real lesson: a finite cache is defined less by its size than by what it chooses to forget.
Why a cache must throw things away
After this lesson you’ll be able to diagnose a cache that’s working correctly but still quietly destroying database performance — and you’ll know the one-line config fix that most teams miss.
A cache is a bounded amount of fast memory standing in front of a much larger slow store. The store holds everything; the cache holds a tiny, hot subset. Two mechanisms decide what’s in that subset. Eviction answers “memory is full and I need room — what do I delete?” TTL (time to live) answers “this value has been here a while — when do I stop trusting it?” They solve different problems: eviction is about space (capacity pressure), TTL is about time (staleness). A production cache almost always uses both.
The reason it matters is the working set — the set of keys actually being accessed in a given window. If your cache is at least as big as the working set, almost every read is a hit and eviction rarely fires. If the cache is smaller than the working set, you thrash: every read evicts something you’ll need soon, and your hit ratio collapses. So the first question is never “which policy?” — it’s “is the cache big enough to hold the working set, or am I trying to fit an elephant in a shoebox?”
The eviction policies and what each assumes
Which entry should a cache sacrifice when memory is full? The answer is not obvious — and choosing wrong silently kills your most valuable data. Each policy is a bet about the future encoded in a rule about the past:
- LRU (Least Recently Used) evicts the entry untouched for the longest. The bet: recency predicts reuse — what you used lately you’ll use again. This matches most real access patterns (temporal locality) and is the sensible default.
- LFU (Least Frequently Used) evicts the entry with the fewest accesses. The bet: popularity predicts reuse. LFU shines when a stable set of hot keys should survive a flood of one-off reads — a one-time scan can’t evict your perennially-hot keys the way it can under LRU.
- FIFO (First In, First Out) evicts the oldest inserted entry regardless of use — exactly the hook’s bug, because the most valuable item is often inserted early and stays valuable. FIFO ignores access entirely; it’s rarely the right cache policy.
- Random evicts an arbitrary entry. It sounds dumb, but it’s cheap (no bookkeeping) and surprisingly robust — it can’t be defeated by a pathological scan the way strict LRU can. Redis’s
allkeys-randomexists for exactly this.
Together, these policies exist to maximize the chance that a key you’ll need soon is still in memory when you need it. When you see a hit ratio collapsing under a scan-heavy workload, that’s the signal to look at whether LRU’s “recency” bet is actually the right one for your access pattern.
▸Why this works
Why do real caches approximate LRU/LFU instead of implementing them exactly? Because perfect LRU means maintaining a precise access-ordered linked list on every single read — extra writes and locking on the hottest path. Redis instead samples: on eviction it picks a few random keys (default 5) and evicts the best one by the policy’s metric, an approximation that gets very close to true LRU/LFU for a fraction of the cost and no global ordering structure. The senior point is that eviction accuracy is itself a tradeoff against eviction speed — and on a cache doing a million reads a second, the bookkeeping cost of “perfect” can exceed the benefit of the better choice. Approximate-LRU is the default precisely because the approximation is cheap and nearly as good.
TTL: bounding staleness, and the jitter trap
TTL attaches an expiry to a key: after N seconds it’s treated as absent, forcing a reload from the source. As the previous lesson showed, TTL is how cache-aside bounds its staleness — it’s the floor under “how wrong can this value get.” Short TTLs mean fresher data and more misses (more DB load); long TTLs mean fewer misses and staler data. You tune TTL per data class: seconds for a fast-moving feed, hours for a rarely-changing config.
But a uniform TTL hides a sharp edge. If you warm 10,000 keys at startup all with ttl=3600, they all expire in the same second an hour later — and at that instant every one of them misses simultaneously, stampeding the database with 10,000 concurrent reloads. The fix is TTL jitter: add a small random spread (e.g. 3600 ± 300s) so expirations scatter across a window instead of detonating together. Jitter is one line of code and it prevents a self-inflicted thundering-herd you’d otherwise debug at 3 a.m.
Hit ratio is the metric, and a miss costs more than you think
The number that tells you whether a cache is working is the hit ratio: hits ÷ total reads. A cache at 99% hits and one at 90% hits sound close, but they are not: the miss rate went from 1% to 10%, a 10× increase in load on the database behind the cache. Because the backing store is sized assuming the cache absorbs most reads, a hit-ratio drop from 99% to 90% can take the database from comfortable to on-fire. This is the asymmetry that makes caching dangerous: the cache hides how fragile the system underneath it is, and a hit-ratio regression (a bad deploy that lowers it, an eviction storm) hits the database 10× harder than the percentage change suggests.
A miss is also not free in latency: as AWS notes for lazy loading, each miss is three trips (check cache, read DB, write cache), so the user on a miss waits the full backend path plus the cache round trips. Hit ratio, miss penalty, and working-set fit are the three numbers; cache size in GB matters only insofar as it changes them.
▸Common mistake
The dangerous misconfiguration is the maxmemory-policy. Redis defaults to noeviction: when memory is full, writes start failing rather than evicting anything — which is correct for a durable store but catastrophic for a cache, where you wanted it to drop cold keys and keep serving. Teams deploy Redis as a cache, leave the default, and one day every SET errors under memory pressure. For a pure cache you want allkeys-lru (or allkeys-lfu) so any key can be evicted; volatile-lru only evicts keys that have a TTL, which silently fills memory with your TTL-less keys until it can’t evict at all. Pick the eviction policy deliberately, and never run a cache on noeviction.
Your cache hit ratio quietly drops from 99% to 90% after a deploy. The cache latency looks fine. Why is the on-call paged about the database melting down?
At startup you warm 50,000 keys into Redis, each with exactly ttl=600. The service runs fine for ten minutes, then the database spikes hard every ten minutes. What's the cause and the one-line fix?
The default eviction policy in real caches approximates _______ — evict the entry untouched for the longest — because it bets that recency predicts reuse, which matches the temporal locality of most real access patterns, and because sampling a few keys approximates it cheaply on the hot read path.
- 01Distinguish eviction from TTL, and tie both to the working set.
- 02What does each eviction policy bet, and why is LRU the default?
- 03Why is hit ratio the metric, and what are the TTL jitter and maxmemory-policy traps?
A cache is finite, so it must forget — by two independent mechanisms. Eviction handles space pressure (memory is full, delete something): LRU bets recency predicts reuse and is the default, LFU bets on popularity and protects a hot set from one-off floods, FIFO ignores access and can evict your most valuable early item (the hook’s bug), and random is cheap and scan-resistant. Real caches approximate LRU/LFU by sampling rather than maintaining a perfect order, trading accuracy for speed on the hot path. TTL handles time (bound the staleness the lazy patterns permit), but a uniform TTL detonates all warmed keys at once — so you jitter it. The number that actually matters is the hit ratio, not the cache size in GB: a drop from 99% to 90% hits is a 10× increase in database load, because the backing store is sized assuming the cache absorbs nearly everything. And the misconfiguration that takes a cache down is the maxmemory-policy — Redis’s noeviction default makes writes fail under pressure, which is wrong for a cache; pick allkeys-lru/lfu deliberately. Everything here exists to keep the working set on the fast path; the next lesson scales this beyond one node. Now when you see a periodic database spike that lines up with your cache warm-up interval, you’ll know immediately: synchronized TTL expiry, fixed by one line of jitter.
Practice
Start at the top. Tasks go easiest → hardest: recall a fact, apply it to a case, then a senior-level stretch. Open one, attempt it, then reveal.
Something unclear?
Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.