Incident postmortems: the leak, the hang, the thundering restart — and the craft of blameless analysis

The leak: RSS climbs 40 MB/h, OOMKilled every ~30 hours

Detection came late and from the wrong signal: a restart-count graph, not a memory alert. The pattern that names the bug class: RSS grows linearly with traffic, no plateau, until the container limit. Linear-with-traffic growth means live references accumulating per request — fragmentation plateaus, and garbage gets collected; only something reachable grows forever.

Root cause: @lru_cache(maxsize=None) on a pricing function whose arguments included the request’s filter tuple. Near-unique keys per request meant the cache almost never hit — it just grew. The worker lives for weeks; nothing ever evicts; gc is irrelevant because every entry is strongly referenced by design. The proof tool is tracemalloc — snapshot at two points in time, diff by line:

import tracemalloc
tracemalloc.start(25)                  # keep 25 frames per allocation

snap1 = tracemalloc.take_snapshot()    # hour 1 — trigger via an admin endpoint
# ... hours pass ...
snap2 = tracemalloc.take_snapshot()    # hour 5
for stat in snap2.compare_to(snap1, "lineno")[:5]:
    print(stat)
# pricing/cache.py:41: size=2.9 GiB (+2.7 GiB), count=38M (+36M)
#   the lru_cache dict — keyed on (user_id, query, tuple(filters))

Mitigation vs prevention: max_requests recycling stops the OOMKills tonight — honestly labeled a stopgap. Prevention is structural: bounded caches (maxsize= a number you can defend, or a TTL cache where staleness is acceptable), an RSS-per-worker alert that fires at the trend (40 MB/h for 3 hours) rather than the cliff, and a review rule: maxsize=None plus request-shaped keys never passes.

The hang: every async worker frozen, CPU near zero

Detection: p99 goes vertical, every request times out, and the counterintuitive signal — CPU near zero. A spinning loop would burn CPU; a frozen one is parked in a syscall. The tool that turns this from speculation into evidence is py-spy dump — it attaches to a live pid without restarting or instrumenting:

$ py-spy dump --pid 1
Thread 0x7F4A (active): "MainThread"
    recv (ssl.py:1233)
    ...
    get (requests/api.py:73)
    fetch_rates (helpers/fx.py:12)      # three helper layers down
    convert (handlers/checkout.py:88)   # an async def — blocking right here

Root cause: a sync requests.get crept into an async handler behind three layers of helpers — the author of fetch_rates never saw an async def in their file. The loop is cooperative and single-threaded: a blocking syscall never yields back to the scheduler, so every coroutine on that worker — hundreds in flight — waits with it (the python/11 bridging rule, now as an outage). Every worker that touches the path freezes the same way; the fleet dies one worker at a time as traffic finds the route.

Prevention: a lint/import rule banning sync HTTP clients (requests, raw urllib3) from async modules — enforced in CI, not in review memory; the await asyncio.to_thread(...) escape hatch where a sync library is unavoidable; an async-native client (httpx) as the default; and a load test that exercises the slow upstream, because this bug is invisible at p50.

The thundering restart: a healthy binary, a full outage

Root cause chain: a release added a 2-minute ML model load at startup. The liveness probe — initialDelaySeconds: 30 — started failing before the load finished; Kubernetes killed the pod mid-load; the replacement began the same 2-minute load; killed again. CrashLoopBackOff across the fleet, from a binary with zero bugs. Detection had a signature worth memorizing: restart loops with clean logs — no traceback, no error, the process was killed, it did not crash — plus probe-failure events in kubectl describe.

Prevention: a startup probe — liveness is suspended until it passes, which exists precisely for slow-boot apps; lazy or background model loading with readiness off until warm (traffic waits, nobody dies); and a boot-time budget asserted in CI — a test that fails the build when startup crosses N seconds, because boot time regressions arrive silently, one dependency at a time.

The craft: writing the postmortem

The three incidents share a skeleton, and the skeleton is the craft. Timeline with timestamps — when did it start, when detected, when mitigated; the detection gap (six weeks, for the leak) is itself a finding. Blameless is not politeness, it is data quality: the engineer who flipped the flag will tell you exactly what happened only if the document cannot be used against them. Five-whys aimed at the system: not “why did the engineer add lru_cache” but “why did our review, our alerts, and our limits all allow unbounded growth to ship and run for six weeks”. Action items are code or config changes with owners and deadlines — an alert created, a probe added, a lint rule merged. “Be more careful” appears in zero competent postmortems; if a human slipping causes an outage, the system was already broken.