awesome-everything RU
↑ Back to the climb

Observability

Trace context propagation in logs

Crux Every log line emitted inside a traced request must carry trace_id and span_id. Without them, the log-to-trace pivot fails and observability degrades to guesswork.
Your altitude — climbing toward senior
ZeroJuniorMiddleSenior
You are at senior altitude — in orbit
◷ 12 min

An engineer finds a log line showing a 5xx error from a checkout service. She searches for more context. The log line has no trace_id. She cannot open the trace, cannot see which downstream call failed, and cannot reconstruct the request flow. She has a symptom and nothing else. Without trace_id, structured logs are an island.

Why trace_id is the most valuable field

A log line with a trace_id can be clicked into the corresponding distributed trace. That trace shows every span across every service that touched the request: per-span timing, dependency calls, error spans, upstream and downstream context. The log line is no longer an island — it is a node in the full observability graph.

Without trace_id, the same log line carries a symptom but no path to the cause. The only option is to correlate by timestamp and hope that no other requests interleaved — which fails the moment the service handles more than one concurrent request.

The OTel Logs Data Model encodes this relationship directly: every log record carries TraceId, SpanId, and TraceFlags as first-class fields, inherited from the active span at emit time. This is not optional metadata — it is the structural join key.

Without trace_idWith trace_id
Log line is an island. You know something happened, not why.Log line links to the full distributed trace for the request.
Correlation requires timestamp guesswork across services.One click opens spans across all services involved.
Concurrent requests interleave in log search results.Query by trace_id isolates exactly one request end-to-end.
Incident investigation: minutes to find root service.Incident investigation: one pivot to the failing span.

How trace context reaches the log line

The trace context — an immutable struct holding the current trace_id, span_id, and trace flags — lives in the language’s request-scoped execution context. The logger reads from it at emit time, automatically.

  • Node.js: the OTel SDK stores context in AsyncLocalStorage. Pino’s mixin callback (or the @opentelemetry/api context.with() mechanism) reads the active span from the store at emit time. Every log emitted within a request handler inherits the trace_id with no per-call-site boilerplate.
  • Go: context.Context carries the active span. The slog handler or the zap logger reads from the context passed to the handler function. Unlike Node, context is explicit — the ctx must be threaded through the call chain.
  • Python: contextvars.ContextVar holds the active span (Python 3.7+). Structlog’s merge_contextvars processor reads the active span at render time.
  • JVM: the OTel Java agent auto-instruments the thread-local context. Log4j2’s ThreadContext or Logback’s MDC carries the trace_id automatically when the agent is attached.

In each case the pattern is the same: the logger has a hook (mixin, processor, handler) that consults the active execution context and injects trace_id and span_id into the log record without the calling code having to pass them explicitly.

Why this works

The reason trace_id auto-injection matters is volume. A typical service emits thousands of log lines per second across dozens of call sites. Requiring engineers to pass {trace_id, span_id} manually to every log call is impractical — it is forgotten during refactors, omitted in third-party libraries, and skipped under time pressure. Auto-injection at the logger level is the only reliable mechanism at scale.

Async context propagation pitfalls

The logger reads context from the active execution context. Anything that breaks the execution context propagation breaks the trace_id on the log line. This is the most common source of trace_id = "00000000000000000000000000000000" entries.

Node.js pitfalls: setTimeout, setImmediate, fire-and-forget Promises, and worker threads can each drop AsyncLocalStorage if not wrapped in context.bind() or context.with(). A callback scheduled with setTimeout(() => { logger.warn(...) }, 0) fires outside the original request context if the span is not explicitly propagated into the callback.

Python pitfalls: asyncio.create_task propagates contextvars correctly since Python 3.7. But threads spawned via threading.Thread and ProcessPoolExecutor subprocesses do not inherit the context automatically — the context must be captured and restored manually.

Go pitfalls: there is no ambient context propagation in Go. The context.Context must be threaded explicitly through every function in the call chain, including goroutines. A goroutine launched without the parent context (go func() { logger.Warn(...) }()) will not carry the trace_id unless the context is explicitly passed in.

Detection: query your log backend for trace_id IS NULL or trace_id = "000000000000..." as a fraction of total volume. If the fraction is above 1%, something is dropping context on a hot path. Pair this with a CI test that exercises async paths (fire a request, wait for an async callback to complete, assert that its log line carries the inbound trace_id).

Quiz

A Node.js service observes that approximately 5% of log lines have trace_id = '00000000000000000000000000000000'. Most likely cause?

Quiz

Why is passing trace_id manually to every log call not a reliable solution at scale?

Order the steps

Order the steps to investigate a 5xx error using the log-to-trace pivot:

  1. 1 Query logs: service=checkout level=error in the last 15 minutes
  2. 2 Find a matching log line; copy its trace_id field
  3. 3 Open the tracing backend; paste the trace_id to load the full distributed trace
  4. 4 Identify the failing span — the one with error=true or the longest duration
  5. 5 Navigate to the service that owns the failing span
  6. 6 Query logs for that service filtered by the same trace_id to see its full log context
  7. 7 Cross-reference: log message confirms the error type; span confirms the timing and call chain
Recall before you leave
  1. 01
    How does trace context auto-injection work in Node.js with pino and the OTel SDK?
  2. 02
    In Go, why does context propagation not work automatically for goroutines, and what is the fix?
  3. 03
    What is the production signal for async context propagation regressions, and how do you set up detection?
Recap

trace_id and span_id are the join keys that connect a log line to the full distributed trace. Without them, a log line is an island — you have a symptom but no path to the cause. The mechanism: the OTel SDK stores the active span in the language’s request-scoped execution context (AsyncLocalStorage in Node, context.Context in Go, contextvars in Python, thread-local MDC in JVM); the logger reads from it at emit time via a mixin, processor, or handler hook, injecting trace_id and span_id automatically. The failure mode: async boundaries (setTimeout, unbound Promises, goroutines without explicit context, Python threads) drop the execution context, producing all-zeros trace IDs. Detection: monitor the trace_id IS NULL fraction per service; alert above 1%; add a CI test that asserts async log lines carry the inbound trace_id.

Connected lessons
appears again in167
Continue the climb ↑OTel Logs Data Model and audit logs as a subsystem
shortcuts expand
search
K
prev piece
k
next piece
j
cycle tier
t
this menu
?
sources3
expand
  1. 01
  2. 02
  3. 03

Trademarks belong to their respective owners. Editorial reference only.