Cardinality as a cost driver: labels, PII, exemplars, and sampling

Cloudflare 2022: a global outage was preceded by Prometheus servers OOMing under cardinality from a new label on the request-duration metric. The fix landed in 90 minutes — but the post-mortem mandated a per-team cardinality budget and a CI check that rejects new label dimensions over a threshold. The alert came from Prometheus’s own meta-monitoring, not from the services it was supposed to watch.

The math of cardinality

Every Prometheus metric series lives in the TSDB head block at roughly 3 KB of RAM. The number of series is the product of all label-value cardinalities:

series count = |route| × |method| × |status_class| × |service| × |region|

For a service with 200 routes, 5 methods, 4 status classes, 50 pods across 3 regions:

200 × 5 × 4 × 50 × 3 = 600,000 series
600,000 × 3 KB = ~1.8 GB just for this one metric

Now add user_id with 100k active users:

600,000 × 100,000 = 60 billion series

This crashes a 16 GB Prometheus server in seconds. The TSDB cannot index that many series, and the append path serializes on the head mutex.

The cost in hosted backends

At Datadog’s ~$0.05 / custom metric / host / month (2024 pricing), an unbounded user_id label that grows to 1M series adds ~$50k/month overnight for one careless label.

The cardinality-to-cost linearity is what makes this a security and financial incident, not just a performance issue.

The PII security angle

A naive Errors counter labelled by error_message or stack_trace publishes exception text into the metrics scrape, which is often less access-controlled than the application database. If the message contains user input — “could not find user alice@example.com” — that PII lands in a metrics backend that the entire engineering org can read.

Real incident: a payments service in 2021 leaked customer phone numbers via a poorly-named failed_phone label. The post-mortem mandated a global pre-commit hook that flags any new label named with a known-PII pattern.

Label audit rule: label by error class (auth_failed, db_timeout, parse_error), never by error content. Audit label names as a security review item, not just a performance review item.

Exemplars: the bridge between metrics and traces

If you cannot put trace_id in a metric label (unbounded cardinality), how do you jump from a p99 spike to the slow request that caused it? Exemplars.

Prometheus 2.32+ and OpenTelemetry’s histogram implementation both support exemplars: sampled trace IDs attached to individual histogram observations. When histogram_quantile shows p99 at 800 ms, clicking the spike in Grafana reveals the exemplar — a trace ID from a request that landed in that bucket. One click jumps to the full span tree.

# HELP http_request_duration_seconds
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{le="0.2"} 14324 # {trace_id="abc123"} 0.183
http_request_duration_seconds_bucket{le="0.4"} 14329

The exemplar trace_id="abc123" is attached to the specific observation 0.183, not added as a label to the metric. Cardinality stays flat; drilldown is preserved.

Aggregation vs sampling

RED + USE metrics are pre-aggregated — they summarise across all requests or all wall-clock time without sampling. A histogram’s bucket counts are incrementally updated; you never throw away an observation.

Traces are the opposite: sampled (typically 0.1–5%) because each trace carries the full request path with all spans. The senior pattern:

• Pre-aggregate RED + USE at the source — 100% coverage, bounded storage.
• Sample traces: head-based at 5% for cost, tail-based at 100% for errors and slow requests (duration > SLO target) — so the rare slow path always has a trace.
• Exemplars bridge the two: the metric shows the spike (aggregate), the exemplar points to a specific trace (sample).

The four-signal stack — RED metrics, USE metrics, sampled traces, sampled profiles — composes if and only if they share label keys (http.route, service.name, status_class). OpenTelemetry’s semantic conventions formalise these join keys.