Head sampling and tail sampling: deciding which traces survive

Your service handles 10,000 requests per second. Storing every trace is prohibitively expensive. But if you sample randomly at 1%, the one slow request that triggered the customer complaint has a 99% chance of being silently discarded.

Head-based sampling

The keep/drop decision is made at trace start — encoded in the traceparent trace-flags 01 (sampled) bit. The decision is propagated to all downstream services, which honour it by default.

Common strategies:

• Probabilistic: keep N% of traces (1–5% typical). Simple, predictable, scales linearly with traffic.
• Rate-limiting: keep at most K traces per second regardless of traffic volume.

Cost: cheap — the decision is made once at the root span, before any work is done. Unsampled requests generate no spans at all, saving CPU, network, and storage.

Drawback: the decision is blind to the outcome. A slow or error-prone request that happens to be in the unsampled 99% is invisible in the tracing backend. If an incident hits 0.5% of traffic and you sample at 1%, you will keep roughly half the incident traces — but you might keep none if the incident is brief.

The sampled flag: when the flag is 01, downstream services record and export their spans. When it is 00, OTel SDKs create spans internally but do not export them by default. This is consistent sampling: either the whole trace is kept or none of it, never a fragment. A downstream service may override the incoming flag (for example, always sample its own errors), but partial overrides produce fragmentary traces that are nearly useless for debugging.

Tail-based sampling

The OTel Collector buffers all spans for a trace-id for a configurable decision window (30s–5min), then decides whether to keep the trace based on policies:

• Error present → keep 100%.
• Duration > threshold → keep 100%.
• Specific attribute (e.g. user.tier=premium) → keep 100%.
• Probabilistic top-up → keep 1% of the rest for baseline visibility.

Advantages:

• Catches every error trace, even at 0.1% traffic rate.
• Catches every slow trace above the latency threshold.
• Provides the kind of selectivity that makes tail sampling the dominant pattern at high-traffic services.

Cost: the collector must hold every trace’s spans in memory until decision time.

Memory model: active_traces × avg_spans_per_trace × bytes_per_span. At 50,000 in-flight traces × 100 spans × 1 KB per span = 5 GB RAM. The decision window directly controls the RAM footprint.

Load-balancing exporter requirement: with multiple collector replicas, random span distribution scatters a trace’s spans across different instances. Each instance only sees fragments and cannot make a correct keep/drop decision. The solution is a load-balancing exporter that hashes by trace-id and routes all spans for one trace to the same collector instance. This is mandatory for tail sampling to work correctly.

The hybrid pattern (dominant in production)

Head-sample at 100% (every request enters the pipeline), then tail-sample by policy:

• Error traces → 100% keep.
• Latency > 99th percentile threshold → 100% keep.
• Everything else → 1% probabilistic.

This gives the volume control of head sampling and the selectivity of tail sampling, at the cost of one piece of additional infrastructure: the tail-sampling Collector tier with load-balancing exporter and sufficient RAM.

At 10k req/s with 30s decision window, 10 spans/trace, 1 KB/span: 10,000 × 30 × 10 × 1,024 B = ~3 GB collector RAM. Doable with 4–8 collector replicas.