OTel architecture: one SDK, four signals, one wire format

Before OpenTelemetry, wiring a new service into your observability stack meant four separate SDK installs, four collector configs, and four query languages. Switching vendors meant re-instrumenting everything. OTel collapsed all four signal types into one SDK and one wire protocol. Switching backends is now a config change.

The full stack architecture

An OpenTelemetry-based production stack looks the same in 2026 regardless of vendor.

Application layer. Every service runs one OTel SDK (or OTel auto-instrumentation) that emits four signals:

• Logs — high-cardinality event records with a trace-id field attached.
• Metrics — low-cardinality counters and histograms; histogram exemplars carry the trace-id of a representative request.
• Traces — per-request causal graphs as OTLP span batches.
• Profiles — function-level CPU and allocation samples; each sample carries the trace-id of the active request.

Every signal carries the same trace-id when applicable, so cross-signal joins are possible at query time.

Collector layer. A local OTel Collector agent (DaemonSet on every node, or sidecar per pod) receives all four signal types via OTLP gRPC (localhost:4317). It batches and ships to a gateway tier. The gateway tier performs:

• Tail sampling on traces — buffer spans until the trace closes, then keep 100% of errors and slow traces plus a small random baseline.
• Metric aggregation.
• Optional PII scrubbing on log fields and profile symbol filtering.
• Routing signals to one or more backends.

Backend layer. Any combination of: Tempo (traces), Prometheus / Mimir (metrics), Loki (logs), Pyroscope (profiles) for the self-hosted Grafana stack; or Datadog, Honeycomb, New Relic for vendor-managed. OTLP is accepted by every major backend as of 2024–2026. Switching the backend is a collector exporter config change; the application code is unchanged.

A concrete OTel collector config

The Node checkout service ships all four signals via a single Collector:

# otel-collector.yaml (DaemonSet on every node)
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
processors:
  batch:
    send_batch_size: 512
    timeout: 10s
  tail_sampling:
    policies:
      - { name: errors, type: status_code, status_code: { status_codes: [ERROR] } }
      - { name: slow,   type: latency,     latency: { threshold_ms: 2000 } }
      - { name: rest,   type: probabilistic, probabilistic: { sampling_percentage: 5 } }
exporters:
  otlphttp/tempo:         { endpoint: http://tempo:4318 }
  prometheusremotewrite:  { endpoint: http://prometheus:9090/api/v1/write }
  loki:                   { endpoint: http://loki:3100/loki/api/v1/push }
  otlphttp/pyroscope:     { endpoint: http://pyroscope:4040/ingest }
service:
  pipelines:
    traces:   { receivers: [otlp], processors: [tail_sampling, batch], exporters: [otlphttp/tempo] }
    metrics:  { receivers: [otlp], processors: [batch], exporters: [prometheusremotewrite] }
    logs:     { receivers: [otlp], processors: [batch], exporters: [loki] }
    profiles: { receivers: [otlp], processors: [batch], exporters: [otlphttp/pyroscope] }

Switching from this self-hosted Grafana stack to Datadog means changing the four exporter endpoint URLs. The service code is unchanged.

Why tail sampling requires a separate processor

Head sampling decides at trace start whether to keep the trace. If it keeps only 5% randomly, then 95% of errors and slow traces are discarded — lost exactly when you need them.

Tail sampling buffers all spans in memory until the trace closes (or a decision window of ~10 seconds expires), then decides. The canonical production policy:

• Keep 100% of traces with any ERROR span.
• Keep 100% of traces with total duration above the slow threshold (e.g. 2 s).
• Keep a random 5% baseline of everything else.

This drops trace volume 90%+ while preserving every interesting trace.

The trace-id as load-bearing join key

Trace-id is the identifier that makes the four-signal stack feel like one tool rather than four silos.

• Logs emit it as a structured field (trace_id: "abc123").
• Metrics emit it as histogram exemplars (a concrete trace-id attached to a bucket sample).
• Traces are keyed by it natively.
• Profiles attach it to each stack sample.

With consistent trace-id propagation (W3C traceparent through every service call), a query for “everything related to this specific request” returns log entries, metric exemplars, the full trace tree, and profile samples — all joined by the same hex string. Without it the four signals are four silos and the engineer manually correlates by timestamp + service name + best guess.

The 128-bit randomness ensures global uniqueness. The W3C standardisation ensures it survives vendor and language transitions. The OTel SDKs handle propagation automatically across HTTP, gRPC, queues, and async calls.