Production propagation failures, span links, and platform design

GitHub ran a propagation regression for a quarter where 50% of internal traces were orphans. The tracing dashboards showed traces the whole time. Nobody noticed until an engineer spot-checked the orphan-span rate in a routine review.

Real production propagation failures

Uber 2019: a partial OTel rollout caused 30% of traces to break at the boundary between instrumented and uninstrumented services. Postmortem mandated a “no service ships without W3C propagation” gate enforced in CI. The pattern: instrumented services emit perfect spans; uninstrumented services emit orphans; the two groups show different trace depths in dashboards, but there is no automated alert on the boundary failure.

GitHub 2022: a custom HTTP client wrapper bypassed OTel’s hooks and silently dropped traceparent across half their internal services for a quarter before someone noticed the orphan-span rate had risen from 1% to 50%. The fix was a single line to wrap the client in the OTel-aware version. The lesson: custom wrappers are the most common propagation gap in mature services. The fix is always a one-liner; finding it takes a quarter.

Slack 2023: tail-sampling Collectors OOMed and took down their tracing pipeline during a major incident — precisely when tracing was most needed. Postmortem added num_traces caps and a separate always-keep tier for high-priority traces. A monitoring gap: OTel Collector health metrics were not on any SLO dashboard.

Datadog 2024 customer report: a large Java workload had a thread pool that didn’t carry context across submitted tasks, so 80% of background-task traces were orphans. Fix: switch to a CurrentTraceContext-aware executor. The bug was present for months; it was discovered during a quarterly orphan-rate review.

The shared pattern: propagation bugs are silent. The dashboards keep showing traces. The only detection mechanism is a metric on orphan-span rate, and that metric must be on a dashboard and ideally on an alert — it is never surfaced automatically by OTel defaults.

Observability for propagation itself

The essential propagation health metrics:

Span links: when the parent-child tree breaks down

The parent-child model assumes linear causality: A calls B, B calls C. This breaks in three scenarios:

1. Batch processing: a consumer pulls 1,000 messages from Kafka and processes them in one batch. There is no single meaningful “parent” — 1,000 incoming trace contexts feed one batch span.
2. Fan-in: multiple parallel sub-jobs converge at a join point. Each sub-job is a child of its own branch; the join point has multiple causal contributors.
3. Async follow-ups: the originating request finishes and returns to the user, but spawns follow-ups that execute hours later. The original request’s context is closed; the follow-ups need a causal link without being children of a dead span.

OTel’s span-links solve all of these: a span declares additional SpanContext references it is causally related to but does not descend from. Tracing backends visualise links as dotted lines alongside the solid parent-child tree.

The senior pattern: any trace longer than 30 seconds or wider than 100 spans is a candidate for span-link refactoring. Split the long workflow into sub-traces where each sub-trace fits within the tail-sampling decision window, and use links to preserve causal lineage. This keeps traces small, keeps the sampler happy, and preserves the investigation chain.

Long-running traces and the 30-minute problem

Tail samplers have decision windows of 30s–5min. A batch job running for 30 minutes emits spans long after the decision window closes; the late spans look like orphans to the sampler.

Two production patterns:

1. Break the work: split long workflows into sub-traces linked via span-links, each fitting in the decision window. Clean architecture, correct by construction.
2. Backend late-span support: Tempo, Honeycomb, and Datadog all support late-arriving spans up to 24h after trace start. Skip tail sampling for long traces; use head sampling at 100% for batch workloads. Practical retrofit for legacy batch jobs.

The decision window is the lever to adjust when batch workloads break tail sampling. Tuning it upward increases Collector RAM; the right answer is usually to break the work.

# Query: trace_id == "4bf92f3577b34da6a3ce929d0e0e4736"
# Result: 7 spans total

#  service           span_id           parent_id           duration   status
1  api-gateway       1a2b3c4d5e6f7890  -                   18ms       OK
2  auth              7890abcdef123456  1a2b3c4d5e6f7890    14ms       OK
3  inventory         abcdef1234567890  1a2b3c4d5e6f7890    1200ms     OK
4  payment           fedcba0987654321  -                   80ms       OK    # ORPHAN
5  postgres-client   1111222233334444  fedcba0987654321    55ms       OK
6  email-job         5555666677778888  -                   240ms      OK    # ORPHAN
7  audit-log         9999aaaabbbbcccc  -                   12ms       OK    # ORPHAN

# Also separate orphan traces with single spans:
# trace_id 9981a... payment service, 78ms
# trace_id ab32c... email-job service, 280ms
# trace_id ff8e1... audit-log service, 14ms