Design a metrics & alerting system

Requirements

A metrics and alerting platform answers one operational question continuously: is the system healthy right now, and if not, who do we wake up? That splits into functional and non-functional needs.

Functional. Collect numeric measurements (counters, gauges, histograms) from thousands of services and hosts. Store them as time series — a stream of (timestamp, value) points identified by a metric name plus a set of key/value labels (http_requests_total{service="checkout", status="500", region="eu"}). Let engineers query them with an expressive language (rate, aggregation, percentile estimation). Evaluate alert rules continuously and fire to a notifier (PagerDuty, Slack). Render dashboards that re-query the data on every refresh.

Non-functional. Ingest must be cheap per point because there are enormous numbers of points. Queries over recent data must be fast (dashboards refresh every 15–30 s; an SLO breach must be visible in seconds). The system must stay up especially when the monitored system is failing — monitoring that dies in an incident is worse than useless. And retention is tiered: raw resolution for days, coarse rollups for years.

The dominant design force, threaded through everything below, is cardinality — the count of distinct label-set combinations. It, not raw request volume, decides whether the system survives.

Estimation

Size it before drawing boxes. Suppose 5,000 hosts, each exporting ~1,000 metrics, scraped every 15 seconds.

active series      = 5,000 hosts × 1,000 metrics            = 5,000,000 series
ingest rate        = 5,000,000 series ÷ 15 s                ≈ 333,000 samples/sec
samples per day    = 333,000 × 86,400                        ≈ 2.9 × 10^10 samples/day
on disk (raw)      = ~2.9 × 10^10 × ~1.5 bytes (compressed)  ≈ 43 GB/day
in-memory index    = ~5,000,000 series × ~few hundred bytes  ≈ a few GB of label index

Two numbers matter. Samples per second (~333K/s) sets the write path — comfortable for a single well-tuned node, trivially shardable beyond. Active series (5M) sets the memory footprint, because the database keeps an in-memory index from labels to series plus the most recent chunk of each series. Now imagine adding a user_id label with a million values: 5M series becomes 5 billion, the index no longer fits in RAM, and the box dies — exactly the hook. Compressed storage (Gorilla-style delta-of-delta timestamps and XOR’d float values, ~1.5 bytes/sample versus 16 raw) is what makes the disk number tolerable; nothing makes unbounded cardinality tolerable.

High-level design

The shape is a collection path (get samples in), a storage core (the TSDB), and two read paths that share it: ad-hoc queries for dashboards, and a scheduled rule evaluator that runs the same query language to compute alerts and pre-aggregated “recording rules.” A separate alert manager deduplicates, groups, and routes firing alerts so one bad deploy doesn’t page twelve people twelve times. Old data is compacted and downsampled into cheap object storage for long retention.

Deep dive

Pull vs push collection

There are two ways to get samples into the system, and the choice ripples through the whole design.

Pull (Prometheus): the server holds a list of targets and scrapes each one’s /metrics HTTP endpoint on a schedule. The server controls the sampling rate, a failed scrape is itself a signal (the target is up/down metric comes free), and service discovery (from Kubernetes, Consul) keeps the target list current. The cost: the server must reach every target on the network, and short-lived jobs that vanish between scrapes need a push gateway as a workaround.

Push (StatsD, OpenTelemetry, InfluxDB line protocol): clients send samples to a collector. This suits ephemeral jobs, serverless, and clients behind NAT that the server can’t reach. The cost: the server can’t tell “no data because healthy and idle” from “no data because the sender died,” you need a separate liveness signal, and a misbehaving client can flood you — backpressure and rate-limiting move to the ingest tier.

Neither is universally right. Pull dominates infrastructure monitoring (stable, discoverable targets); push dominates application events and serverless. Many large platforms run both: pull for the fleet, a push gateway for batch jobs.

Cardinality: the explosion that kills the database

This is the single most important operational fact about metrics systems, and it’s the hook. A time series is uniquely identified by its full label set, so the number of series is the product of the cardinalities of every label:

series for one metric = ∏ (distinct values of each label)

http_requests_total{service, status, region}
  service: 50  ×  status: 8  ×  region: 5   = 2,000 series   ✅ fine

add label user_id (1,000,000 distinct):
  50 × 8 × 5 × 1,000,000              = 2,000,000,000 series ❌ database dies

Every series carries fixed overhead: an entry in the inverted label index, an open chunk in memory for incoming samples, file handles. The active-series count, not the sample rate, is what blows the memory budget — and it grows multiplicatively with each unbounded label. The rules that keep it bounded are non-negotiable senior practice: never put unbounded-cardinality values in labels (user IDs, email addresses, full URLs with query strings, request IDs, timestamps). Those belong in logs or traces, which are built to be searched by high-cardinality keys; metrics are built to be aggregated over bounded label sets. When you genuinely need per-customer latency, you either bucket customers into a bounded tier label, or you answer that question from traces/logs, not from the metrics TSDB.

Downsampling and retention tiers

Nobody needs 15-second resolution for a graph spanning a year — and storing it is wasteful and slow to query. So a compactor runs in the background, merging recent high-resolution blocks and computing downsampled rollups: 5-minute aggregates, then 1-hour aggregates, each retained longer. A typical tiered policy:

raw (15 s)    → kept 15 days   (incident debugging, recent dashboards)
5-min rollup  → kept 90 days   (capacity trends, weekly reviews)
1-hour rollup → kept 2 years   (year-over-year planning)

Downsampling can’t just keep the average — for each window you store multiple aggregates (min, max, sum, count) so a query can still compute a meaningful rate or approximate percentile over the coarse data. Old blocks live in object storage (S3/GCS), not on the hot node’s disk; the query engine reads them on demand and caches results. This is the kappa-style split between a small, fast, in-memory hot tier and a large, cheap, cold tier — and it’s why a “show me last year” query is slower but possible, while a “show me last 30 minutes” query is instant.

Bottlenecks & tradeoffs

The binding constraint is memory for active series, and almost every tradeoff is about defending it. Cardinality is the failure mode; relabeling/dropping at ingest and hard per-target series limits are the defenses. Resolution vs cost is a knob: scrape every 15 s and you see fast spikes but pay 4× the storage of a 60 s scrape — most teams scrape fast and downsample early. Alerting freshness vs query cost is a hidden coupling: alert rules run the same query engine as dashboards, so a heavy recording rule or an expensive dashboard query competes for the same CPU — a slow query path makes alerts late, which is how a monitoring system fails silently during the incident it should be catching. Mature systems pre-compute hot aggregates with recording rules (cheap reads at alert time) and isolate the alerting path from ad-hoc dashboard load. Finally, the system must be more available than what it watches: run it separately (its own cluster, its own region), keep its dependencies minimal, and make sure that when everything else is on fire, the thing watching the fire is still standing.