ISR storms and cold starts: the self-DDoS you schedule for yourself

Anatomy of an ISR storm

By the end of this lesson you will know exactly which decision on your team’s part produced the 8-second p95 — and which single infrastructure change stops it from repeating at the next marketing send. ISR in steady state is a load-smearing machine. Each entry carries its own revalidate window measured from when it was generated; a request after the window serves the stale copy instantly and triggers one background regeneration. Load on the origin is one render per page per window, smeared across time by traffic. Every storm is the same event: something breaks the smearing and synchronizes the invalidations.

Trigger one: the deploy resets the cache. Self-hosted, the Full Route Cache and Data Cache live by default on the filesystem under .next/cache inside the container — a rolling deploy replaces containers, and each new one starts empty. On managed platforms a new deployment starts with only the build-time prerenders seeded; every on-demand entry accumulated since the last deploy is gone. Either way, the long tail you spent days warming evaporates in one rollout. Trigger two: a broad revalidation. A CMS bulk-publish fires a webhook per entry and the handler calls revalidatePath for each; or someone “fixes staleness” with revalidatePath('/', 'layout'), which invalidates every route in the app at once. Trigger three: aligned windows — after any synchronized wipe, all entries regenerate within minutes of each other, so their windows now expire together, and the storm repeats every window until traffic de-synchronizes them.

The arithmetic is what turns this from slow to incident. 30,000 catalog pages, 4 queries each, regenerated within ten minutes is 120,000 queries on a database sized for the steady-state trickle — a self-DDoS with your own render path as the botnet. Worse: after a wipe there is no stale copy to serve, so the stale-while-revalidate cushion is gone — first requests block on the render instead of getting the old page, which is why the storm shows up as user-facing latency and not just DB load. And with replicas it multiplies: three pods with three private filesystem caches means up to three independent renders per path, plus content flip-flop as the load balancer alternates between a pod that regenerated and one still serving the old copy. The ISR mechanics are from the rendering-strategies lesson at the start of this track; the tag-based invalidation surface is in the data-layer unit — this lesson is what happens when both are operated at fleet scale.

Mitigations that change the mechanism

Before you reach for “add more pods,” ask yourself: does adding a pod share the cache or duplicate it? Each real mitigation removes one of the storm’s preconditions; none of them is “add more pods” (more pods with private caches make it worse).

Persist the cache out of process. A custom cacheHandler in next.config moves the Full Route Cache and Data Cache into Redis or another shared store. Two preconditions die at once: the cache survives deploys (new containers attach to the same store, so the long tail stays warm), and replicas share entries (one regeneration per path, no flip-flop). The correctness caveat that bites teams: if the deploy changes a page’s shape, old cached entries are now wrong — include the build id in the key, or revalidate changed sections on deploy through the queue below, deliberately, instead of relying on the accidental full wipe.

Jitter the windows. revalidate: 3600 on every page means one synchronized wipe aligns every expiry forever after. Derive the window per path — 3600 plus a hash of the slug spread over ±600 seconds — and expiries never re-synchronize. This is the cheapest mitigation in the list: one function, no infrastructure.

Queue and cap regeneration. Bulk webhooks must not call revalidatePath inline, once per event. Enqueue the slugs, deduplicate, and let a worker drain the queue at a concurrency sized to the database’s spare headroom — if Postgres has 500 QPS to spare and a render costs 4 queries, the cap is around 100 concurrent regenerations, and a 5,000-SKU bulk publish takes a calm few minutes instead of one hot one.

Serve stale while storming. A CDN in front with stale-while-revalidate and stale-if-error is the outer shield: during the storm users get a bounded-staleness page instead of an 8-second render. State the staleness budget out loud — “during regeneration storms, prices may be up to 10 minutes old” is a product decision, and writing it down is what makes the mitigation legitimate.

Cold starts: what init actually spends

A cold start is everything between “no instance exists” and “your handler runs”: provision and extract the bundle, parse and compile the JavaScript — the dominant, size-proportional term; tens of MB of dependencies cost seconds of CPU before any request logic — boot the framework (route manifests, instrumentation), then dial connections: TLS plus database handshake is 50-300 ms per instance, paid before the first query. A trim route lands its init in the low hundreds of milliseconds; a route that bundles a heavy ORM, an AWS SDK, and a markdown pipeline pays 2-5 seconds.

Cold starts are invisible at p50 and own your p99, and they cluster at exactly two predictable moments. Deploy hour: a rollout replaces every instance, so the whole fleet is cold simultaneously and the p99 is the init histogram for several minutes. Trough-to-burst: overnight scale-to-zero means the 09:00 burst lands on zero warm instances, and because the burst is concurrent, the platform spins up N instances in parallel — N users each pay a full cold start at once, which is the 8-second p95 from the Hook. The honest mitigation list: smaller route bundles (the biggest lever — analyze per-route bundle size, lazy-import the heavy paths, keep giant SDKs out of shared modules), provisioned concurrency (a floor of always-warm instances; it genuinely works and costs a small always-on fleet — price it against the SLO instead of pretending it is free), and regional placement near the data (a cold start far from Postgres pays the cross-region RTT on top, and then every warm request keeps paying it). Measure cold starts as a first-class metric: cold-start rate (share of invocations with an init phase) plus an init-duration histogram — alerting on p99 latency alone tells you about the storm after the users do.