Metrics and cardinality: the cost model of a time-series database

A team adds a customer_email label to their request_total counter to track per-customer error rates. Within two hours the Prometheus server runs out of memory and 15 minutes of alerts go missing. The label looked harmless.

How a metric is stored

The canonical model is Prometheus. A metric is identified by a (metric_name, label_set) pair, where the label set is a small fixed collection of key=value pairs. Every unique combination of label values creates one time series — a stream of (timestamp, value) samples, typically scraped every 15 s.

Four standard metric types:

• Counter — monotonically increasing integer. Only a rate-of-change (per second) is meaningful. Example: http_requests_total.
• Gauge — instantaneous reading that can go up or down. Example: memory_used_bytes, queue_depth.
• Histogram — distributes observations across pre-defined buckets for percentile queries. Example: http_request_duration_seconds with buckets [0.01, 0.05, 0.1, 0.5, 1.0, 5.0].
• Summary — pre-computed per-replica percentile. Rarely used for new instrumentation because it cannot be aggregated across replicas.

The compute model is pre-aggregation at the source: the application increments counters in-process, the scraper ships them to the TSDB, and the TSDB stores deltas and serves queries by walking pre-rolled-up buckets. This is why metric queries return in tens of milliseconds across years of history: the heavy lifting happened at write time, not read time.

Cardinality: the central cost driver

Each unique combination of label values creates a separate time series in the TSDB. The math is multiplicative:

• route (20 templates) × method (5 verbs) × status_class (4 classes) = 400 series — fine.
• Same metric + user_id (100 k active users) = 100 k × 20 × 5 × 4 = 40 million series — Prometheus OOM on 16 GB.

Prometheus stores roughly 3 KB per active series in the head block. At 40 M series that is 120 GB, far beyond any commodity machine’s memory.

Both Grafana’s published guidance and the Prometheus instrumentation documentation say the same thing: label only with values from a small fixed set — route templates not full URLs, status classes not status codes, customer segments not customer IDs. Anything with unbounded values belongs in logs (free cardinality) or traces (sampled), not metrics.

The exemplar bridge

When a histogram records a slow observation, the client can attach one exemplar — a trace_id from the request that produced the observation. Grafana renders exemplars as dots on the histogram heatmap; clicking a dot opens the corresponding trace.

Exemplars (standardised in Prometheus 2.32, early 2022) solve the gap that defined the pre-exemplar era: “metrics say something is slow, but I have no example request to drill into.” They make it possible to keep user_id out of metric labels — cardinality stays bounded — while still being able to navigate from an aggregate spike to one concrete request via the trace.