CPU profiles and the heap-snapshot leak hunt

CPU profiling: find the wide frame

When a service is slow or pegging a core, you do not guess — you sample. A CPU profile records which function is on the stack at each sampling interval, so the function that appears most often is where time actually goes. The fastest way in Node is the built-in flag --cpu-prof: it writes a .cpuprofile file on exit that you open in the Chrome DevTools Performance tab. There you read a flame graph where each box is a stack frame and width = self time — the widest frames are your hot path, full stop. Depth is just call nesting; ignore it and chase width.

# writes isolate-*.cpuprofile on clean exit; open in Chrome DevTools → Performance
node --cpu-prof --cpu-prof-dir=./prof server.js

# the classic V8 tick processor: a text report, no GUI needed
node --prof server.js
node --prof-process isolate-*.log > processed.txt

--prof plus node --prof-process gives you the same sampled data as a text tick report — handy on a headless box where you can’t open DevTools. Higher-level tools wrap this: 0x generates an interactive flame graph from one command, and clinic flame does the same with a guided workflow. All of these are sampling profilers (snapshot the stack every N microseconds, cheap, ~1-2% overhead) as opposed to instrumentation (wrap every call, exact counts, but so heavy it distorts the very timings you’re measuring). For production triage you almost always want sampling. The classic find: a regex or a deep JSON.parse shows up as one fat frame eating 40% of self time — optimize it, cache it, or move it off the main thread to a Worker.

Heap snapshots: the three-snapshot leak hunt

A CPU profile won’t find a leak; for that you photograph the heap. A heap snapshot is a complete graph of every live object and who points to whom. Capture one with v8.writeHeapSnapshot() from inside the process, trigger one on demand with --heapsnapshot-signal=SIGUSR2 (then kill -USR2 <pid>), or take one from the DevTools Memory tab. One snapshot tells you what’s big now; it can’t tell you what’s leaking. For that you use the three-snapshot technique.

import v8 from "node:v8";

// snapshot 1 — baseline, after warm-up
v8.writeHeapSnapshot("./snap-1.heapsnapshot");
await exerciseSuspectPath({ times: 5000 }); // hammer the suspected route
// snapshot 2 — after the load
v8.writeHeapSnapshot("./snap-2.heapsnapshot");
await exerciseSuspectPath({ times: 5000 }); // do it again
v8.writeHeapSnapshot("./snap-3.heapsnapshot");

Load all three into the Memory tab, switch the dropdown to Comparison, and use “Objects allocated between snapshots”: anything whose count only ever grows from snap 1 → 2 → 3, and never drops after a GC, is your leak. A stable workload should reach a sawtooth — allocate, collect, repeat — so the steady upward staircase is the tell. The next column is the one that matters most: not shallow size (the object’s own bytes) but retained size — the memory that would be freed if this object were freed, i.e. everything only it keeps alive. Sort by retained size, pick the constructor whose count grew, and follow its retainer path (the chain of references holding it) straight to the cache, array, or listener that won’t let go.

GC, the heap limit, and what “leak” actually means

Before you reach for --max-old-space-size, ask yourself: does memory return after a full GC, or does the floor keep climbing? The answer tells you whether you have a provisioning problem or a leak — and the fix is completely different.

V8’s heap is generational. New allocations land in new space, collected by Scavenge — fast, frequent, copying survivors out. Objects that survive a couple of Scavenges get promoted to old space, collected by the slower Mark-Sweep-Compact. The practical definition of a leak falls right out of this: old space (and RSS) that climbs across full GCs and never comes back down. A healthy process sawtooths; a leaking one staircases. Watch it directly with --trace-gc, which prints one line per collection with before/after sizes.

node --trace-gc server.js
# [12345:0x...] 4210 ms: Mark-Sweep 1850.2 (2048.0) -> 1844.9 (2048.0) MB
#                                    ^ after a full GC it barely dropped → leaking

The hard ceiling is --max-old-space-size=<MB>. On 64-bit Node the old-space default is large but bounded (historically ~2 GB; it varies by platform and version), and a real workload with a big legitimate heap may need it raised. But raising it is a tuning knob, not a leak fix — as the hook shows, doubling the limit just doubles the time to OOM. The diagnostic question is always the same: does memory return after a full GC? If yes, you’re under-provisioned, raise the limit. If no, you have a leak, take snapshots.

The leak-triage runbook

When memory climbs and you get OOM-killed, the moves are mechanical, not magical. Confirm it’s a leak (old space doesn’t drop after a full GC via --trace-gc). Take a baseline snapshot, exercise the suspected path hard, snapshot again, repeat for a third. Use the comparison view’s “allocated between snapshots” to find the constructor whose count only grows. Sort by retained size, open the retainer path, and walk it to the root reference — almost always an unbounded cache, a forgotten listener, a fat closure, or a module global. Then make the reference go away: add LRU eviction or a TTL to the cache, removeListener/off on teardown, null the closed-over object. Re-run the snapshot diff to prove the count is now flat. A regression test that asserts steady-state RSS after N iterations keeps it from coming back.

// before: leaks one entry per request, forever
const cache = new Map();
function handle(req) {
  cache.set(req.id, expensiveResult(req)); // never evicted → OOM in hours/days
}

// after: bounded — old keys become unreachable and GC reclaims them
import { LRUCache } from "lru-cache";
const cache = new LRUCache({ max: 10_000, ttl: 60_000 });
function handle(req) {
  cache.set(req.id, expensiveResult(req)); // evicts oldest past 10k entries
}