SLO platforms and the 90-day rollout

A team re-derives the same 14.4x burn-rate PromQL for the sixth time this month — different service, same math, different typo on line 4. The recording rule was wrong. The alert never fired during the incident. Nobody noticed until the postmortem.

Why hand-rolling is a trap

The canonical MWMBR setup for a single SLO requires:

• 6 recording rules (ratio_rate per window: 5m, 30m, 1h, 6h, 3d, and a slow-burn variant)
• 3 alert rules (14.4x page, 6x page, 1x ticket), each using AND between two windows
• 2 dashboard panels (burn rate over time, budget remaining)

Hand-writing this per service across 80 services means 480 recording rules and 240 alert expressions. Any of the 14.4 * 0.001 multipliers could be wrong (wrong SLO target, off-by-one window). The error is silent — the recording rule computes a value, the alert fires or not, and nobody knows the math was wrong until a real incident doesn’t page.

SLO platforms exist to solve this: you describe the SLI and target in a YAML declaration, and the platform generates the PromQL.

The major platforms

A minimal Sloth declaration looks like:

version: "prometheus/v1"
service: checkout
slos:
  - name: availability
    description: "Checkout requests succeed"
    sli:
      events:
        error_query: sum(rate(http_requests_total{status=~"5..",job="checkout"}[{{.window}}]))
        total_query: sum(rate(http_requests_total{job="checkout"}[{{.window}}]))
    objective: 99.9
    alerting:
      name: CheckoutAvailability
      page_alert:
        labels:
          severity: page
      ticket_alert:
        labels:
          severity: ticket

Sloth reads {{.window}} as a template variable and emits all six recording rules with correct window values. The burn rates (14.4x / 6x / 1x) and the AND logic between windows are injected automatically. You declare the SLI expression once; the platform ensures correctness.

The 90-day SLO rollout pattern

SLO adoption fails at the cultural layer more often than the technical one. A common rollout pattern:

Weeks 1–2 — single journey, single team: Pick the most important user journey (checkout, login, the API endpoint customers pay for). Define the SLI at the API gateway level. Set a conservative target (start with 99% even if you think you can do 99.9% — you’ll learn the baseline quickly). Instrument counters, add the Sloth declaration, verify the recording rules appear in Prometheus.

Weeks 3–4 — verify SLO and budget: Watch the SLO for two weeks. Is the baseline error rate what you expected? Is the SLI correlating with real user reports? Adjust the target if baseline is much better (tighten) or worse (too tight SLOs create constant freeze culture, loosen if necessary). Define the error budget policy draft but don’t sign it yet.

Weeks 5–8 — alerting and policy sign-off: Enable MWMBR alerts. Add the SLO dashboard. Run a fire drill: deliberately break the service for 5 minutes and verify the 14.4x alert fires within a few minutes, pages the on-call, and clears within 5 minutes of fix. Sign the error budget policy with director-level approval. Communicate the policy to all stakeholders.

Weeks 9–12 — rollout to 5–10 more services: Replicate the pattern across other services. Each team uses the same Sloth YAML template. The platform team owns the Sloth/Pyrra generator; product teams own their SLO targets. Hold a first quarterly SLO review: which SLOs fired alerts that mattered, which were noise?

After 90 days — culture check: Survey the on-call teams: are they acting on SLO alerts or ignoring them? Review: are feature freezes actually happening when budgets are exhausted? If not, re-run the policy communication. The 90-day mark is also when to look at whether SLOs are covering the right journeys — gaps are common on the first pass.