Failure modes and engineering practice: cardinality budgets, PII, and sampling

At 14:00 a developer added customer_email as a label to http_requests_total. By 14:03 Prometheus had OOM-killed itself: 9.24 million series, 14.3 GB of RAM, restart count 1. The WAL replay lagged 73 seconds; every alert fired stale or not at all. One label, three minutes, regional monitoring down.

Production failure modes per signal

Each signal type has a characteristic way of failing. Knowing the failure mode tells you where to put the guardrails.

Each failure mode has a detection metric. Senior teams monitor the monitoring tier as carefully as the product tier.

Cardinality budgets as engineering practice

In any 1.0 metrics stack, cardinality discipline is an engineering practice, not a one-time choice at instrumentation time.

The Cloudflare 2022 pattern: every team owns a cardinality budget (e.g. 100k active series across all services in the team’s namespace). CI checks flag any PR that introduces a new label name exceeding the threshold. The budget is reviewed quarterly, resized based on actual usage. The tool suite: Grafana’s mimirtool, Datadog’s Custom Metrics Usage view, Prometheus’s prometheus_tsdb_head_series and count by (__name__) ({__name__=~".+"}).

The cultural payoff is that engineers think about label cardinality at write time — not after the OOM at 03:00.

The Prometheus OOM case study. At 14:02 prometheus_tsdb_head_series jumped from 824k to 9.24M in two minutes — 11x growth. At ~3 KB per active series, the head block needed ~27 GB; the server had 16 GB. Memory hit 14.3 GB (89% of limit) and the kernel OOM-killed Prometheus. On restart, the WAL replay was 412 segments, lagging 73 seconds behind scrape time.

Immediate mitigation (under 5 minutes, no deploy required): add metric_relabel_configs with action: labeldrop and regex: customer_email to the Prometheus scrape config of the affected service. Confirm prometheus_tsdb_head_series_created_total rate falls back toward baseline within one scrape interval. Memory does not reclaim until the next 2-hour block compaction; restart Prometheus only if memory continues to climb.

Durable fix: (a) remove customer_email from the metric; (b) replace with a bounded customer_segment label (free/pro/enterprise = 3 values); (c) use exemplars to attach a sampled trace_id so per-customer drill-through still works without paying the cardinality cost; (d) add a CI check (cardinality-lint.ts) that fails builds when new label names match a deny-list regex (email, user_id, request_id, session_id, customer_id); (e) add a Prometheus alert: rate(prometheus_tsdb_head_series_created_total[5m]) > 1000 pages on-call before OOM.

PII risk on metric and log surfaces

Both metric labels and log fields can leak PII into a less-audited datastore than the source database.

Real incidents:

• A payments company in 2021 leaked customer phone numbers as a metric label (failed_phone="+1..."). Prometheus scraped the metric to all environments including dev; the phone numbers were visible in dashboards to the whole engineering org.
• A SaaS company in 2023 ingested raw request bodies into logs (including session tokens) for two months before a customer noticed in a support thread. The logging pipeline had no PII filter.
• A marketplace in 2024 emitted user search queries as a query span attribute, exposing demographic-segment information to the BI team who had Jaeger access but not production DB access.

Mitigations:

1. Allow-list for labels and span attributes. Deny by default. Every new metric label or span attribute requires explicit approval in a reviewed config file checked in alongside the code.
2. Redaction in the logging pipeline. Vector and Fluent Bit both ship with regex redactors for common PII patterns. The pattern list (email, phone, ssn, token, password, session_id) should be maintained in a security-reviewed repo alongside the cardinality deny-list.
3. Audit dashboards. A dashboard that shows high-cardinality string fields per signal — sorted by distinct-value count — catches leaks before they reach a customer’s awareness.

Treat label review and field review as a security gate, not a performance gate.

OTel sampling configuration in practice

The OTel Sampler API provides head-based decisions at the SDK and tail-based at the collector.

SDK-level samplers:

• AlwaysOn — 100% sampling. Use only in dev/test or for very low-volume critical paths.
• AlwaysOff — 0% sampling. Useful for internal health-check routes where traces add no signal.
• TraceIdRatioBased(p) — samples p fraction of root traces. Generates a random decision correlated with the trace_id; must be uncorrelated with request properties (see below).
• ParentBased(root_sampler) — honours the parent’s sampling decision from the incoming traceparent header. Downstream services follow the root’s decision automatically via the W3C sampled flag.

Collector-level tail sampling (TailSamplingProcessor):

Policies applied after the trace completes. Common production layout:

policies:
  - name: errors-policy
    type: status_code
    status_code: {status_codes: [ERROR]}
  - name: slow-traces-policy
    type: latency
    latency: {threshold_ms: 1000}
  - name: baseline-policy
    type: probabilistic
    probabilistic: {sampling_percentage: 2}

The errors-policy and slow-traces-policy keep 100% of interesting traces. The baseline-policy keeps 2% of the rest, giving low-traffic services coverage when there are no errors.

The Elastic 2024 post-mortem on head sampling. Naive head sampling under-represents errors and slow tails because the sampling decision is made before the trace has any outcome. The post-mortem also flagged a subtler problem: TraceIdRatioBased generates a random decision from the trace_id. If the trace_id is not truly random (e.g., derived from a hash of the request path), the sampling decision correlates with request properties — slow endpoints may always be sampled or always dropped as a cohort. Production rule: use ParentBased(TraceIdRatioBased(0.2)) at the edge with a cryptographically random trace_id, and layer tail policies at the collector for the interesting cases.