Incident to enforcement: SLO burn to verified fix in 35 minutes

An SLO burn alert fires at 10:47 am. Without continuous profiling and distributed traces, the on-call spends the afternoon guessing. With them, she has a root cause and a fix deployed by 11:22 am. The 35-minute window is not luck — it is the previous investment in observability paying off under pressure.

The incident: /checkout p99 burns the SLO

At 10:47 am, an alert fires: “checkout SLO p99 burn rate 14x — 12 hours of error budget consumed in 6 hours.”

The on-call engineer opens the unified dashboard. No deploys in the last hour. Traffic at baseline. This is not a spike — something changed in a downstream service.

Step 1: Observe — 60 seconds

Prometheus: /checkout p99 jumped from 240 ms to 1.4 s. Started approximately two hours ago. RPS flat. Error rate flat. Pure latency degradation.

RUM: client-side LCP unaffected. This is a server-side problem, not a bundle regression.

Step 2: Profile — 90 seconds

Pyroscope filtered to /checkout over the last two hours:

runtime.scanobject:   22%   (baseline: 4%)
runtime.mallocgc:      9%   (baseline: 2%)
encoding/json.Marshal: 24%  (baseline: 4%)
handlers.Checkout:      6%
pgx.Query:             12%

GC frames jumped from 6% to 31%. JSON Marshal jumped from 4% to 24%. Something is producing much more garbage and serialising much larger payloads than before.

Step 3: Classify — 60 seconds

Allocation-bound (family: GC / piece 04), with the serialisation overhead pointing at an unusually large response being produced, not just processed. This is the N+1 family pattern (piece 05) applied to inter-service payloads: something upstream started returning bulk data where a summary was sufficient.

Amdahl on the combined GC + Marshal frames (55%): 1 / (1 - 0.55) = 2.2x. That would bring p99 from 1.4 s to 636 ms — still above the 200 ms SLO. The root cause is not just “GC is hot”; it is whatever is causing the GC and Marshal pressure.

Step 4: Trace correlation — 5 minutes

Open Tempo, filter to slow /checkout traces in the last two hours. The waterfall shows:

[span] handle_request          1820ms
  [span] auth_check               5ms
  [span] order_query             340ms   (baseline: 12ms)
  [span] inventory_check         850ms   (baseline: 80ms)
  [span] payment_call            280ms   (baseline: 90ms)
  [span] serialise_response      340ms   (baseline: 8ms)

Every downstream span and the serialisation step degraded by the same multiplier. This is the pattern of an upstream service returning a response that is much larger than expected — every consumer step inflates proportionally.

Check deploy history: inventory-service rolled out at 10:31 am — 16 minutes before the SLO burn started. Diff: added include_full_sku_details: true to the /inventory response. Previously returned SKU IDs only; now returns full SKU objects. Response payload: 8 KB → 85 KB per call.

The /checkout service receives 85 KB, deserialises it (more allocations), selects 0.1% of the data it needs, then serialises its own response including the inflated inventory data. All three cost centres — GC, Marshal, downstream latency — trace to a single cause.

Step 5: Fix — 15 minutes

Two parallel actions:

1. Roll back the inventory-service deploy via the deploy console.
2. Add a defensive check in /checkout: reject any inventory response over 64 KB with an error before processing.

Both ship within 15 minutes.

Step 6: Verify — 10 minutes

Pyroscope after rollback: scanobject returns to 4%, mallocgc to 2%, json.Marshal to 4%. Profile baseline restored.

Prometheus: /checkout p99 at 235 ms within 5 minutes of the rollback completing. SLO burn rate drops to 0.1x.

Both the local profile AND the headline metric confirmed.

Step 7: Enforce — the sprint after

Three enforcement actions, all taken in the following sprint:

A. PR gate: inter-service response size. Any PR to any service that changes a response schema must pass a contract test asserting the body size does not exceed 2x the current median. PRs that grow a response by more than 2x require explicit SRE review.

B. Production alert: per-endpoint payload size p99. Add a metric tracking p99 response body bytes per downstream call. Alert if it grows more than 50% week-over-week. First time this class of regression fires, the alert routes to the responsible team’s on-call before the SLO burns.

C. Runbook entry. “p99 spike on /checkout with all spans degraded proportionally → check upstream deploys in the last 2 hours for response-schema changes. Check inventory-service specifically.” The next on-call resolves this in under 5 minutes instead of starting from scratch.

Total elapsed from page to fix: 35 minutes. Total elapsed to enforcement: one sprint (1 week). Engineer-hours: incident response + postmortem + enforcement: 6 hours. Without continuous profiling + traces: estimated 4 to 8 hours for the incident alone.