Sampling strategies: head, tail, and parent-based

At 10k requests per second, recording 100% of traces fills a backend in hours and generates a five-figure monthly bill. But if you drop 99% of traces, you lose exactly the errors and slow requests you need for debugging. Sampling is the tradeoff between cost and fidelity.

Head sampling: decide early, cheaply

The SDK’s Sampler runs on the very first span of a trace (the root span) and decides whether to record the trace. The decision is propagated to downstream services via the W3C traceparent header’s sampled flag.

AlwaysOn — records every trace. Full fidelity, maximum cost.

AlwaysOff — records nothing. Useful for a canary or a service under load test.

TraceIdRatioBased(p) — records a random p-fraction based on a hash of the trace_id. Deterministic across services: if two services both use TraceIdRatioBased(0.1), they independently make the same decision for the same trace because the hash is of the trace_id, which is shared across the request. At p=0.1, 10% of traces are recorded.

ParentBased — defers to the parent span’s sampling decision (via traceparent.sampled). When an upstream service decides to sample, all downstream services follow. When upstream does not sample, downstream typically does not, unless the local sampler overrides. Prevents inconsistent partial traces.

Production combination: ParentBased(root=TraceIdRatioBased(0.2)). The root span samples 20% of traces. All downstream spans inherit the decision. The result: either the full trace is recorded or nothing is — no partial traces with orphaned child spans.

Tail sampling: decide late, expensively

Tail sampling runs in the Collector after all spans of a trace have arrived and the trace is complete (or close enough). It can examine the full trace context: final status code, total latency, presence of error spans anywhere in the tree, business attributes on any span.

Typical tail sampling policies:

• status_code=ERROR — keep 100% of traces with any ERROR span
• latency > 1000ms — keep 100% of slow traces
• probabilistic 1% — keep 1% of everything else as a baseline

The combination covers all interesting traffic (errors, slowness) and provides a cost-bounded baseline for normal traffic.

The sticky-routing constraint

Tail sampling requires that every span of a trace arrive at the same Collector gateway instance. If spans of one trace land on two different instances, neither can evaluate the full trace context and the sampling decision is wrong.

The OTel Collector’s loadbalancing exporter on the agent tier solves this: it hashes by trace_id and routes deterministically to the same gateway pod for all spans of a trace.

Tradeoff: the gateway becomes stateful. A pod restart or scale-up event reshuffles the hash ring and loses in-flight traces. Production mitigations:

• Pre-warm new pods before they enter the load-balancing ring
• Use conservative scaling policies (avoid frequent scale-up/down during traffic peaks)
• Size the gateway’s num_traces buffer for peak rate × decision_wait × safety factor (~2x)

Numbers

• Tail sampling buffer: ~1-2 GB per 50k active traces (100 spans each at ~1 KB/span)
• decision_wait: 30-60 seconds (must exceed p99 trace duration)
• num_traces: size for peak_rate × decision_wait × 2 — at 2,000 traces/sec and 30s window: ~120k
• Head sampling baseline: 10-20% in production
• Tail sampling keeps: 100% errors + 100% slow (threshold 1s) + 1-5% baseline
• Net retained: ~3-5% of total traffic, 100% of interesting traffic