Multi-window multi-burn-rate alerting: why AND beats OR

An on-call team uses a single alert: “fire if the error rate over the past hour exceeds 1.44%.” The pager fires at 2 AM. The engineer logs in. The issue cleared itself 55 minutes ago — but the 1-hour window still contains the bad data. The engineer stays awake for nothing.

Why naive SLO alerts fail in two ways

The SRE workbook classifies six approaches to SLO alerting. Five fail in predictable ways.

Approach 1 (raw error rate > SLO threshold): fires hundreds of times per day on tiny blips. A single bad request batch triggers the page; teams learn to ignore the pager — the classic alert fatigue death spiral.

Approach 5 (single burn-rate threshold, e.g. 14.4x over 1h): better — catches real outages — but the reset is slow. After the incident is resolved, the 1-hour window still contains 55 minutes of bad data. The alert keeps firing for nearly an hour post-fix. Engineers cannot tell if their change helped.

Approach 6 (MWMBR) is the only one that balances detection latency, noise resistance, and recovery latency. It is the production default in Google, Datadog, Grafana, Splunk, Honeycomb, Sloth, and Pyrra.

The dual-window trick

The key insight: combine a long window (noise resistance) AND a short window (recovery resolution):

• Long window (e.g. 1h): confirms the burn is sustained, not a transient spike
• Short window (e.g. 5m): confirms the burn is still happening right now

The AND logic:

• If only the long window is high: the incident probably ended; the short window cleared
• If only the short window is high: a brief spike that has not yet accumulated into a sustained burn
• If both are high: a real outage that is both severe and still active — page

When the incident resolves, the short window clears within 5 minutes. The alert resets within 5 minutes instead of within 55 minutes. Detection time and reset time are both bounded.

The canonical MWMBR configuration (99.9% SLO)

Prometheus implementation

# Recording rule: fast-request ratio over a 1h window
record: job:slo_latency_fast:ratio_rate1h
expr: |
  sum(rate(http_request_duration_seconds_bucket{le="0.2"}[1h]))
  /
  sum(rate(http_request_duration_seconds_count[1h]))

# Page alert: 1h AND 5m both at >14.4x burn
alert: SLOLatencyBurnFast
expr: |
  (
    (1 - job:slo_latency_fast:ratio_rate1h) > (14.4 * 0.001)
    and
    (1 - job:slo_latency_fast:ratio_rate5m) > (14.4 * 0.001)
  )
labels:
  severity: page
annotations:
  summary: "Latency SLO burning at >14.4x over 1h AND 5m"

The 14.4 * 0.001 is burn_rate × (1 − SLO) = 14.4 × 0.001 = 0.0144 — the error rate threshold corresponding to 14.4x burn at 99.9% SLO.