Error budget policy, latency SLOs, and composite journeys

Every service in the checkout path shows 99.9% availability on its own dashboard. The user-facing checkout success rate is 99.5%. Both numbers are correct — the team is monitoring the wrong layer.

The error budget policy: more than a rule

An error budget policy is a written, signed document that specifies what happens when the budget is exhausted. Without the signatures, it is a suggestion that pressured on-call teams ignore during release crunches. With a director-level signer, it is a covenant.

Mandatory contents:

• Deployment freeze trigger: “when budget reaches 0, halt feature deploys except P0 fixes and security patches”
• Postmortem trigger: “if a single incident burns >20% of the budget, a postmortem is mandatory”
• Freeze exit: “focus on reliability work until budget regenerates above 50% remaining”
• Escalation path: “if the freeze lasts >2 weeks, escalate to VP/CTO”

Out-of-scope exclusions (events that do not burn the budget):

• Load test and penetration test requests (tagged by header or IP range)
• Known bot and scanner traffic
• Requests during planned maintenance windows
• Requests rejected by rate limiting at the edge

These exclusion categories must be explicit in the policy document and enforced as label filters in the SLO platform — otherwise every postmortem becomes an argument about whether the incident “should count.”

Latency SLOs: buckets, not quantiles

An availability SLO is trivially a ratio of counts: 5xx_count / total_count. A latency SLO requires expressing “X% of requests under Y ms.” This sounds like a percentile, but it is implemented as a counter:

latency_sli = http_request_duration_seconds_bucket{le="0.2"} / http_request_duration_seconds_count

The histogram bucket at the SLO threshold gives this counter directly — no histogram_quantile needed, and no estimation error contaminating the budget. This is why the RED-Duration histogram must have a bucket boundary exactly at the SLO threshold: without it, the SLO is unevaluable without approximation.

Multi-threshold latency SLO: “90% of requests under 100ms AND 99% of requests under 500ms” catches tail-latency hiding behind a single percentile. A service might pass the 99%-under-500ms threshold while hiding a degraded tail that fails the 90%-under-100ms check. Production-grade SLO platforms support multiple thresholds per SLO via separate SLI ratios joined with worst-of logic.

Composite SLOs: the multiplication problem

A checkout journey through API gateway → auth → inventory → payment → database means a request is “good” only if all five services were good. Each service independently at 99.9% gives a journey ceiling of 0.999⁵ ≈ 99.5% — not 99.9%. With dependent failures (shared dependencies, regional incidents), the ceiling is lower still because failures cluster in time.

The fix: add a journey-level SLI at the API gateway — count successful checkout completions / total checkout attempts, not per-service 200 codes. Per-service SLOs become diagnostic context; the headline is the journey SLO. This is the layer users actually experience.

Finding the dominant contributor: when the journey SLO is red and per-service dashboards are green, pull per-service failure rates and rank them. Pareto applies: 80% of journey failures typically come from 1–2 services. Fix those first; they give maximum leverage per engineering-hour.