Detecting stampedes and designing TTL for production

The observability fingerprint

Cache stampedes leave a distinctive signature in metrics:

• DB query rate: a sawtooth pattern — near zero between TTL boundaries, a sharp spike at each boundary. The spike width equals the rebuild duration.
• Periodicity: spikes recur at intervals matching the TTL. TTL=60 s → spikes every 60 s. TTL=300 s → spikes every 5 minutes.
• p99 latency: spikes at the same intervals as the DB query rate.
• cache_miss_total rate: sharp periodic increases at boundaries instead of a smooth low baseline.

Without these instrumented, a stampede looks like a general “DB slowness” incident with no obvious cause.

Minimum-viable dashboard

You can deploy every mitigation from the previous lessons and still get paged at 3 am — because you protected the wrong keys. What tells you which key is stampeding, how bad it is, and whether your fix is working? Six metrics, charted in one place.

Six metrics cover all stampede scenarios:

Alert 1: cache_miss_total rate above 5× steady-state → stampede forming. Alert 2: db_query_rate p99 above 10× p50 → sawtooth DB load → boundary spikes. Alert 3: cache_lock_wait_seconds p99 above rebuild duration → lock queue depth growing.

TTL jitter

Single-flight and locks handle per-key stampedes. But what if 1,000 keys all have TTL=300 s and they were all cached at the same time? They all expire together, producing 1,000 simultaneous per-key stampedes — single-flight correctly handles each one, but the sum of 1,000 concurrent rebuilds is a DB spike.

TTL jitter: instead of a fixed TTL, use a random value in a range:

ttl = base_ttl * (1 + jitter_fraction * (rand() - 0.5))
# Example: base=300, jitter=0.25 → TTL in range [225, 375]

A fleet of 1,000 keys with ±25% jitter spreads expiries over 150 s instead of all firing at once. DB load becomes a smooth low curve instead of a spike.

Most cache libraries support jitter natively (Caffeine in Java, Redis via application-level calculation). The default ±15–25% is sufficient for most workloads.

Negative caching

The same stampede shape applies when the database answers “no such row.” If the application does not cache null results, every request for a non-existent key hits the DB — and under high concurrency this is a miss-storm that overloads the DB as fully as a positive-key stampede.

Fix: cache the “missing” sentinel with a short TTL.

# On DB miss:
SET key:missing "" EX 10  # 10 s negative TTL

# On read:
val = GET key
if val == "":
  return NOT_FOUND  # from cache, no DB hit

Short negative TTL (5–30 s) bounds memory churn. The positive TTL can be much longer (60–300 s). Write-through invalidation must delete the negative entry when a real row is inserted.

Security note: without negative caching, an attacker can mint random non-existent keys (random UUIDs in a URL path) to amplify DB load by orders of magnitude — a documented pattern that affected CDN-backed sites in 2024.

Pre-warming after restarts

A cache that restarts cold (deploy, eviction, machine failure) starts empty. Every incoming request misses and hits the DB — a full-traffic origin spike. If restarted at peak traffic, this spike is equal in magnitude to a full stampede.

Pre-warming procedure:

1. Before accepting public traffic, replay the top-N most-accessed keys from an audit log or access log.
2. Warm the cache first, then cut traffic.
3. For blue-green deploys: warm the green cache instance to steady-state before switching the load balancer.

Cloudflare edges pre-warm from neighbouring POPs. Redis-backed services use a startup script reading “top 1,000 keys” from an audit table. The rule: never restart a cache under live traffic without pre-warming.