Metrics, percentiles and SLOs

The four metric types: pick the one that answers your question

A metric is not a log line and not a span. It is a number sampled over time and aggregated server-side — cheap to store, cheap to query, and the thing you build alerts on. Prometheus gives you four types, and choosing wrong is the first mistake.

import { Counter, Gauge, Histogram } from 'prom-client';

// COUNTER: monotonic, only ever goes up (or resets to 0 on restart).
// You never read its raw value — you query rate() over it.
const httpRequests = new Counter({
  name: 'http_requests_total',
  help: 'Total HTTP requests',
  labelNames: ['method', 'route', 'status'] as const, // LOW cardinality — see below
});

// GAUGE: a value that goes up AND down — a snapshot of "right now".
const inFlight = new Gauge({
  name: 'http_requests_in_flight',
  help: 'Requests currently being served',
});

// HISTOGRAM: samples a value (here, duration) into pre-defined buckets,
// so you can compute quantiles SERVER-SIDE across all instances.
const httpDuration = new Histogram({
  name: 'http_request_duration_seconds',
  help: 'Request duration in seconds',
  labelNames: ['method', 'route', 'status'] as const,
  buckets: [0.01, 0.05, 0.1, 0.3, 0.5, 1, 2, 5], // boundaries decide p99 accuracy
});

A counter is monotonic — request totals, error totals. You never graph the raw number; you graph rate(http_requests_total[5m]) to get requests-per-second. A gauge moves both ways — in-flight requests, memory usage, queue depth, connection-pool size: a snapshot you read directly. A histogram drops each observation into a bucket; because the buckets live on the server you can compute histogram_quantile(0.99, …) and, crucially, add the buckets across all your instances first and then take the quantile. The fourth type, summary, computes quantiles client-side in each process — which means you cannot aggregate them: the p99 of three instances is not the average of their three p99s. In any multi-instance service, prefer histograms.

In Nest you wire this with @willsoto/nestjs-prometheus (or raw prom-client), expose a /metrics endpoint for Prometheus to scrape, and record durations in an interceptor that wraps every request — start a timer in intercept, stop it in the tap/finalize of the observable, labelling by method, route template, and status.

RED and the golden signals: what to actually measure

You do not measure everything; you measure the handful of signals that tell you whether the service is healthy. For a request-driven service the RED method is the spine: Rate (requests per second, from the counter), Errors (failed requests per second, the same counter filtered to 5xx), Duration (the latency distribution, from the histogram). Google’s four golden signals add saturation (how full the busiest resource is) to latency, traffic, and errors. USE (Utilization, Saturation, Errors) is the resource-side mirror — apply it to CPU, memory, disk, the connection pool. RED for the request path, USE for the resources it leans on.

Percentiles: the average lies, measure the tail

Here is the core insight the opening incident turns on. The average latency hides the outage that p99 exposes. A mean of 80ms can sit on top of a p99 of two seconds, because the mean is dominated by the bulk of fast requests and a small slow tail barely nudges it. But that tail is not noise — it is real users, and it is exactly the cohort that hits timeouts, fires retries, and amplifies load. An SLO on the average is an SLO on the wrong number.

# PromQL: p99 latency per route, aggregated across ALL instances.
# Note the sum by (...) BEFORE histogram_quantile — this is why you use
# a histogram and not a summary: you add the buckets first, then quantile.
histogram_quantile(
  0.99,
  sum by (le, route) (rate(http_request_duration_seconds_bucket[5m]))
)

So you track p50 (the typical user), p95 and p99 (the tail that hurts), and p999 once your traffic is large enough for one-in-a-thousand to be a meaningful number of people. One caveat: a Prometheus histogram quantile is only as good as your bucket boundaries. histogram_quantile linearly interpolates within the bucket the quantile falls in, so if your p99 lands between a 1s and a 5s bucket, the reported value is a guess somewhere in that gap — possibly very wrong. Put bucket boundaries where you actually care (around your SLO threshold), not on round numbers.

Cardinality: the label that takes Prometheus down

This is the failure mode that kills the monitoring system itself. In Prometheus, every distinct combination of label values is its own time series, stored and indexed separately. http_requests_total{method="GET", route="/users/:id", status="200"} is one series; change any label value and you get another. That is fine when labels are bounded — there are a handful of methods, a few dozen routes, a few status classes, so the product is maybe a few hundred series.

It becomes catastrophic the instant you put a high-cardinality label on a metric: userId, requestId, email, or a raw path that embeds ids. A userId label means one series per user — a million users is a million series per metric, each with its own memory and index entry. Prometheus holds its head series in RAM; cardinality explosion OOMs the server, balloons scrape time and storage, and takes monitoring down exactly when you need it.

// CARDINALITY BOMB — raw path with embedded ids becomes one series PER id.
// /users/1, /users/2, … /users/9999999 → millions of distinct label values.
httpRequests.inc({ method: 'GET', route: req.path, status: '200' });
// e.g. route="/users/8f3a-...-91/orders/4412"  ← unbounded, unique per request

// SAFE — collapse to the route TEMPLATE; the id lives in the value, not the label.
httpRequests.inc({ method: 'GET', route: '/users/:id/orders/:orderId', status: '200' });

The rule: labels must be low-cardinality and bounded — method, route template (not the raw path), status class. When you feel the urge to add a userId or requestId label so you can “drill down” per-user, stop: that id belongs in a trace or a log line, not a Prometheus dimension. The thing you were tempted to put in a label (the user id, the request id) belongs in a trace or a log line, not a metric dimension. The war-story is mundane and common: someone labelled http_requests by full URL including literal /users/{id} ids, the series count exploded, Prometheus ate all its RAM and fell over — and the dashboard went dark mid-incident.

SLOs and the error budget: what you alert on

An SLO is a target: “99.9% of checkout requests complete in under 300ms, measured over 30 days.” Its mirror image is the error budget — the 0.1% you are allowed to miss. That budget is not a failure to avoid; it is a resource to spend. Burn it slowly and you have room to ship features and take risks; burn it fast and you freeze deploys and fix reliability. And you do not alert on the raw error count — a handful of errors at 3am is noise. You alert on the burn rate: how fast you are consuming the budget. A burn rate that would exhaust a 30-day budget in an hour is a page; one that would take a week is a ticket. That is the difference between waking someone up and not.