Observability: distributed traces, USE/RED, and sampling

p95 latency tripled overnight from 80 ms to 240 ms. You have logs from the load balancer, the application, and the database. Three logs, three clocks, three grep sessions. Without distributed traces, “where did the 160 ms go” takes hours. With OpenTelemetry, it takes 30 seconds — look at the span that grew, and you have a suspect.

OpenTelemetry and W3C Trace Context

Modern observability instruments every network hop with a shared identifier. The W3C Trace Context standard defines two HTTP headers:

• traceparent: 00-<16-byte trace-id>-<8-byte span-id>-<flags> — the globally unique trace ID and the current span’s ID.
• tracestate: vendor-specific metadata (Datadog trace ID, Jaeger flags, etc.).

Every component in the request path — browser, CDN edge (via Cloudflare Workers), load balancer, application server, database call, external API call — reads the incoming traceparent, creates a child span with a new span-id, does its work, emits timing data, and passes the same trace-id forward in its outbound request headers.

The result: an observability backend (Tempo, Jaeger, Honeycomb, Datadog APM) can display the entire request as a waterfall of spans — each hop, each dependency, each database query, each external API call — with exact start times and durations. Finding “where did 500 ms go” reduces to looking at the longest span.

Trace 7a8f3c... duration 587ms

span: HTTP request                              0–587ms (587ms)
span: DNS lookup                              0–4ms    (4ms)
span: TCP connect                             4–32ms   (28ms)
span: TLS handshake                           32–67ms  (35ms)
span: HTTP request send                       67–69ms  (2ms)
span: Server processing                       69–512ms (443ms)
  span: Auth middleware                       69–73ms  (4ms)
  span: Database query SELECT users           73–78ms  (5ms)
  span: Database query SELECT user_settings   78–82ms  (4ms)
  span: External API call third-party.com     82–489ms (407ms)
  span: Response serialisation                489–512ms (23ms)
span: HTTP response receive                   512–587ms (75ms)

Propagation through the CDN edge

CDN edges (Cloudflare Workers, Fastly Compute, AWS CloudFront Functions) now support trace propagation. When a request arrives at the edge:

1. Edge reads traceparent from the incoming request.
2. Creates an edge span with its own span-id.
3. Records time-to-first-byte from origin.
4. Passes traceparent (with original trace-id, edge’s span-id) to the origin.

Result: the trace shows CDN edge latency as a distinct span. If the edge span is 5 ms and the origin span is 400 ms, you know to optimise origin, not CDN. Without trace propagation, you see one 405 ms total and guess at the breakdown.

USE and RED operational frameworks

Two disciplined instrumentation frameworks prevent “metric sprawl” — collecting 500 metrics and not knowing which matter.

USE method (Brendan Gregg) for resources (CPU, memory, disk, network interfaces, connection pools):

• Utilization — what fraction of the resource is in use? (CPU 80%, connection pool 95%)
• Saturation — is work queuing because the resource is full? (request queue depth, run queue length)
• Errors — is the resource failing? (TCP errors, disk errors, OOM kills)

RED method (Tom Wilkie) for services (APIs, microservices):

• Rate — how many requests per second?
• Errors — what fraction return errors?
• Duration — what is the latency distribution (p50, p95, p99)?

Using both together. RED tells you what is broken (service error rate spiked). USE tells you why (CPU saturation caused by a CPU-bound handler, or connection pool saturation because the DB is slow). Together they reduce MTTR from hours to minutes.

Sampling strategies

Tracing every request at 1 M req/s produces terabytes of trace data per day — cost-prohibitive. Two strategies balance completeness against cost:

Head-based sampling. Decide at request entry whether to trace — fixed percentage (e.g., 1% of requests). Cheap and deterministic: the trace-id carries the sampling decision propagated to all downstream components. Downside: most errors and slow requests happen in the 99% you did not trace. You have no traces for your worst incidents.

Tail-based sampling. Buffer all spans in memory, decide after seeing the request outcome:

• 100% of requests with error status (4xx, 5xx)
• 100% of requests with duration > threshold (p99 cutoff)
• 0.1% of fast, successful requests (baseline)

Implemented by the OpenTelemetry Collector with a tail_sampling processor. Downside: requires buffering all spans for the decision window (typically 30–60 s), using memory proportional to in-flight requests. At 1 M req/s with 30 s window, that is 30 M spans in memory — manageable with proper sharding.

Adaptive sampling. Adjusts sample rate dynamically based on system load or time of day. During incidents, bumps to 100% for error traces; during quiet periods, reduces to 0.01%.

Right pattern: head-based sampling for steady-state baseline data (cheap); tail-based sampling for errors + slow requests (guarantees traces for what matters); adaptive for cost management under variable load.