Detecting stampedes and designing TTL for production

A team deploys single-flight and locks. Three weeks later an on-call alert fires: DB CPU spiking every 5 minutes. The protection is in place — but it is protecting the wrong keys. The spike is coming from a different key with TTL=300 s that nobody instrumented.

The observability fingerprint

Cache stampedes leave a distinctive signature in metrics:

• DB query rate: a sawtooth pattern — near zero between TTL boundaries, a sharp spike at each boundary. The spike width equals the rebuild duration.
• Periodicity: spikes recur at intervals matching the TTL. TTL=60 s → spikes every 60 s. TTL=300 s → spikes every 5 minutes.
• p99 latency: spikes at the same intervals as the DB query rate.
• cache_miss_total rate: sharp periodic increases at boundaries instead of a smooth low baseline.

Without these instrumented, a stampede looks like a general “DB slowness” incident with no obvious cause.

Minimum-viable dashboard

Six metrics cover all stampede scenarios:

Alert 1: cache_miss_total rate above 5× steady-state → stampede forming. Alert 2: db_query_rate p99 above 10× p50 → sawtooth DB load → boundary spikes. Alert 3: cache_lock_wait_seconds p99 above rebuild duration → lock queue depth growing.

TTL jitter

Single-flight and locks handle per-key stampedes. But what if 1,000 keys all have TTL=300 s and they were all cached at the same time? They all expire together, producing 1,000 simultaneous per-key stampedes — single-flight correctly handles each one, but the sum of 1,000 concurrent rebuilds is a DB spike.

TTL jitter: instead of a fixed TTL, use a random value in a range:

ttl = base_ttl * (1 + jitter_fraction * (rand() - 0.5))
# Example: base=300, jitter=0.25 → TTL in range [225, 375]

A fleet of 1,000 keys with ±25% jitter spreads expiries over 150 s instead of all firing at once. DB load becomes a smooth low curve instead of a spike.

Most cache libraries support jitter natively (Caffeine in Java, Redis via application-level calculation). The default ±15–25% is sufficient for most workloads.

Negative caching

The same stampede shape applies when the database answers “no such row.” If the application does not cache null results, every request for a non-existent key hits the DB — and under high concurrency this is a miss-storm that overloads the DB as fully as a positive-key stampede.

Fix: cache the “missing” sentinel with a short TTL.

# On DB miss:
SET key:missing "" EX 10  # 10 s negative TTL

# On read:
val = GET key
if val == "":
  return NOT_FOUND  # from cache, no DB hit

Short negative TTL (5–30 s) bounds memory churn. The positive TTL can be much longer (60–300 s). Write-through invalidation must delete the negative entry when a real row is inserted.

Security note: without negative caching, an attacker can mint random non-existent keys (random UUIDs in a URL path) to amplify DB load by orders of magnitude — a documented pattern that affected CDN-backed sites in 2024.

Pre-warming after restarts

A cache that restarts cold (deploy, eviction, machine failure) starts empty. Every incoming request misses and hits the DB — a full-traffic origin spike. If restarted at peak traffic, this spike is equal in magnitude to a full stampede.

Pre-warming procedure:

1. Before accepting public traffic, replay the top-N most-accessed keys from an audit log or access log.
2. Warm the cache first, then cut traffic.
3. For blue-green deploys: warm the green cache instance to steady-state before switching the load balancer.

Cloudflare edges pre-warm from neighbouring POPs. Redis-backed services use a startup script reading “top 1,000 keys” from an audit table. The rule: never restart a cache under live traffic without pre-warming.