The performance loop: discipline, not a project

A team fixed their p99 from 1.2 s to 200 ms. They called it done, shipped, and moved on. Six months later p99 is 900 ms again. No single regression — just new features, new libraries, a bigger JSON response from an upstream service. The fix was right. The discipline was missing.

Why performance regresses by default

Every new feature adds bytes, queries, or allocations. Every dependency upgrade ships new code paths. Every schema change can turn a fast query into a slow one. Without a mechanism to catch these additions, performance degrades continuously.

A one-time optimisation has an effective half-life of three to six months. After that window, the accumulated changes from new feature work undo the gains. Teams that treat performance as a project get there — then drift back. Teams that treat performance as a discipline stay there.

The difference is one mechanism: the loop.

The eight-step performance loop

Every performance investigation, regardless of layer, follows the same structure:

1. Observe — a symptom surfaces: SLO burn, RUM regression, user complaint, dashboard alert. This tells you something is wrong, not what.
2. Profile — capture data appropriate to the symptom. CPU flame graph for CPU spikes, allocation profile for memory growth, network waterfall for slow page loads, bundle analyzer for client-side bloat.
3. Classify — name the bottleneck by family: CPU-algorithmic, allocation-bound, cache-bound, lock-bound, I/O-bound (N+1), syscall-bound (batching), JIT-deopt, bundle-bound. Each family has a known fix set.
4. Predict — use Amdahl’s law to estimate how much the headline metric will improve if you fix this hotspot. If the prediction is below your SLO target, this is not the right hotspot; return to step 2.
5. Fix — from the family’s playbook, pick the technique that matches the specific shape of the hotspot. Apply only the predicted change; no scope creep.
6. Verify — re-profile under the same load. Confirm both the local hotspot shrank AND the headline metric improved.
7. Enforce — add a CI gate, alert, or runbook entry that prevents this exact regression from returning.
8. Move on — find the next bottleneck. The loop never ends; it shifts between layers.

The kitchen metaphor

Performance is like cleaning a kitchen, not painting a room. Painting once is fine. A kitchen cleaned once gets dirty as cooking happens; you clean continuously.

Each of the seven pieces in this chapter is a tool: profiler, hot-paths classifier, GC fixer, N+1 detector, batcher, bundle analyzer. None alone keeps the kitchen clean; the loop does.

Bea and Sven’s quarter

Bea joins a team where the service was fast a year ago. Now p99 is 1.2 s, up from 200 ms. Sven walks her through the loop: profile shows GC pressure at 18%, an N+1 in /orders adds 50 queries per request, /dashboard bundle grew 800 KB over six months. No single crisis — three separate slow accumulations.

They run the loop on each bottleneck one at a time: logger allocation fix (week 1), query deduplification (week 2-3), bundle code-split (week 4). After a month, p99 is 280 ms. CI gates keep the work alive through the next quarter of feature shipping.