Native histograms, SLO tie-in, and production failure patterns

GitHub 2023: a SLO alert for code-search latency repeatedly fired without action. Investigation found the alert was on average latency — a bug fix that pushed p99 from 200 ms to 800 ms barely moved the mean. The SLO was reworked to multi-window p99 burn-rate alerts per the Google SRE Workbook pattern. The next incident was caught in under 4 minutes.

Native histograms (Prometheus 2.40+)

Classic histograms require choosing bucket boundaries at instrumentation time. If your service’s latency distribution shifts — say a slow path suddenly dominates the 1–5 s range — you need to redeploy with different buckets.

Native histograms (Prometheus 2.40+, also called exponential histograms in OTel) use a logarithmic scale with dynamically adjusting resolution. Every observation is placed in a bucket sized at ~1% relative precision — e^(2^-scale) width — without any static boundary list. Benefits:

• No manual bucket tuning.
• Uniform p99 accuracy across the latency range (not just near the SLO).
• Similar storage cost to a well-tuned classic histogram.
• Full PromQL compatibility: histogram_quantile works the same way.

OTel’s exponential histograms work identically and are the default for new metric instrumentation in the OTel SDK.

Compatibility note: some older storage backends (Cortex < 1.15, Thanos < 0.32) need version upgrades to read native histograms. Verify the storage tier before switching.

Why averaging p99 across replicas is wrong

Averaging pre-computed percentiles (summary metrics) across replicas is a famous anti-pattern:

Example: replica A serves 100 requests all at 100 ms. Replica B serves 100 requests all at 1000 ms. p99 of A is 100 ms; p99 of B is 1000 ms; their average is 550 ms. The true p99 of the merged 200 requests is 1000 ms (rank 199 out of 200, above the 99th cutoff). The average is off by nearly 2×.

The correct pattern: histograms emit per-replica bucket counts; sum by (le) (rate(...)) aggregates them at query time; histogram_quantile computes the true fleet-wide percentile from the aggregated counts. Accurate within ±half a bucket. No replica isolation.

RED feeds the SLO error budget

An SLO is a promise: 99.9% of requests complete successfully within 200 ms over 30 days. RED’s Duration and Errors are exactly the metrics that populate this promise.

• Availability SLO: 1 - rate(errors[window]) / rate(total[window]) — “99.9% of requests must not return 5xx”
• Latency SLO: histogram_quantile(0.99, ...) < 0.200 — “99th percentile must be under 200 ms”
• Error budget: minutes or request-count remaining before the SLO is breached for the month.

Without RED instrumentation, you do not have an SLO — you have a wish.

Multi-window burn-rate alerting (Google SRE Workbook): alert when the error budget is burning too fast at two windows simultaneously:

• 1-hour window (fast burn): catches sudden spikes (deployment gone wrong).
• 6-hour window (slow burn): catches slow regressions that threshold alerts miss.

A threshold alert on absolute p99 (“fire if p99 > 200 ms”) misses slow regressions (p99 goes from 150 to 190 ms over two weeks, always below the threshold) and fires false positives on brief spikes. Burn-rate alerts catch both.

Production failure patterns in RED+USE

GitHub 2023 (average latency alert): SLO alert on average latency. A regression pushed p99 from 200 to 800 ms; mean moved from 45 to 55 ms. Alert never fired. Fix: switch to p99 burn-rate alerts with 1h short + 6h long windows per the SRE Workbook.

Cloudflare 2022 (cardinality explosion): A new label on request_duration caused Prometheus servers to OOM. Rule-evaluation latency grew from 100 ms to 4+ s, masking service alerts. Prometheus’s own prometheus_tsdb_head_series was the leading signal — but there was no alert on it. Fix: per-team cardinality budget enforced in CI; meta-monitoring on Prometheus’s own RED.

Stripe 2024 (PSI memory pressure invisible to free-RAM): Webhook worker stalled for 11 minutes. Free-RAM dashboards showed 500 MB available. PSI memory full was at 90% — kernel was thrashing reclaim. Fix: migrated saturation alerting from MemAvailable < threshold to PSI-based alerting; next incident caught in 30 seconds.

Discord 2023 (rule-evaluation latency growth): A refactor moved a hot label from one metric to two. Cardinality doubled silently. Prometheus rule-evaluation latency grew from 100 ms to 4 s over two weeks, delaying alerts during a customer-facing incident. Fix: rule-evaluation duration added as a first-class panel in the meta-monitoring dashboard.