OpenTelemetry on AWS: unified signals and the cardinality trap

OpenTelemetry and ADOT: instrument once, export anywhere

The previous lesson was about X-Ray — AWS’s own tracing backend and its agent. The problem with leaning on any single vendor’s agent is lock-in: the day you want to send your traces to Datadog, your metrics to a self-hosted Prometheus, and keep logs in CloudWatch, you are re-instrumenting your entire fleet because the vendor agent only speaks to the vendor. OpenTelemetry (OTel) is the vendor-neutral answer: one open standard for instrumenting the three signals — traces, metrics, and logs — with a single SDK in your code, plus a separate process called the OTel Collector that receives, processes, and exports that telemetry to whatever backend you choose. Your application code emits OTLP (the OpenTelemetry wire protocol); the Collector decides where it lands. Swapping backends becomes a Collector config change, not a code change across every service.

On AWS the supported Collector distribution is ADOT — the AWS Distro for OpenTelemetry: the upstream Collector plus AWS-maintained exporters (X-Ray, CloudWatch/EMF, Amazon Managed Prometheus) and security patches, with AWS support. The Collector runs in one of three topologies, and the choice is a real tradeoff. As an agent/sidecar (one Collector per host or per pod) it is close to the app, adds no network hop, and isolates blast radius — but you run N Collectors and pay that overhead N times. As a central gateway (a load-balanced Collector fleet all services ship to) you centralize sampling, batching, and redaction policy in one place — but it is now a shared dependency you must scale and keep highly available, and it is the natural place to do tail sampling because it sees whole traces. Most serious setups run both: a thin agent for collection, a gateway for policy.

# ADOT / OTel Collector: receive OTLP, batch, then FAN OUT to three backends.
receivers:
  otlp:
    protocols:
      grpc:   # apps push OTLP over gRPC to the local agent
      http:
processors:
  batch: {}   # amortize export calls; never export per-span
exporters:
  awsxray: {}                 # traces  -> AWS X-Ray
  awsemf:                     # metrics -> CloudWatch via Embedded Metric Format
    namespace: Checkout
    dimension_rollup_option: NoDimensionRollup
  prometheusremotewrite:      # metrics -> Amazon Managed Prometheus (AMP)
    endpoint: "https://aps-workspaces.../api/v1/remote_write"
service:
  pipelines:
    traces:  { receivers: [otlp], processors: [batch], exporters: [awsxray] }
    metrics: { receivers: [otlp], processors: [batch], exporters: [awsemf, prometheusremotewrite] }

The tradeoff to state out loud: OTel buys you portability and one instrumentation across every backend, at the cost of operating a Collector (a stateful, memory-bound process you must size and monitor) and living with a spec where some pieces — logs especially — matured later than traces. A turnkey vendor agent skips the Collector entirely; you pay for that convenience in lock-in. The failure mode of getting this wrong is subtle: an under-provisioned gateway Collector silently drops spans under load (the batch queue fills and the exporter sheds), so your traces look complete in dev and have holes in prod exactly when you need them.

Correlating across signals: the trace_id is the join key

The reason you bother unifying the three signals under one standard is correlation: being able to pivot from a metric spike to an example trace to the exact log lines for that one request, in seconds, instead of grepping three disconnected systems. The mechanism is a single identifier — the trace_id — that flows through all three. A trace propagates across service boundaries via the W3C traceparent header, so every hop shares the same trace ID:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             │  └─ trace_id (32 hex) ──────────┘ └ span_id ────┘ └ flags
             └ version

That same trace_id gets stamped into every structured log the request emits, so filtering logs by one trace ID reconstructs exactly what happened on that request. And on the metrics side, exemplars attach a sample trace_id to a metric data point — so when a latency histogram bucket spikes, you click the spike and land on a real slow trace, not a guess. Metric → trace → log, one click each. The glue that makes signals joinable at all is consistent resource attributes: every signal an app emits is tagged with the same service.name, deployment.environment, service.version. Get these inconsistent — one service calls itself checkout, another checkout-svc — and the backend treats them as different services, the service map fractures, and correlation quietly fails with no error. The failure mode here is not a crash; it is dashboards that look fine and lie.

Head vs tail sampling: where you decide what to keep

You cannot store every trace at scale — at 5,000 requests/second that is ~13 billion traces a month, and both the storage bill and the SDK/network overhead make 100% retention absurd. So you sample, typically keeping 1–10% of traces. The senior decision is where the keep/drop choice happens. Head sampling decides at the very start of the trace, at the first service, before any work is done: cheap, stateless, and the decision propagates down the traceparent flags so the whole trace is consistently kept or dropped. Its fatal weakness is that it decides blind — it cannot know this trace will end in a 500 or take 9 seconds, so a flat 5% head sample will, by definition, throw away 95% of your rare error traces, which are the exact ones you needed.

Tail sampling decides after the trace completes, once the outcome is known: keep 100% of traces that errored or exceeded a latency threshold, and sample the boring fast successes down to 1%. The signal is dramatically better — you keep what matters and drop only noise. The cost is structural: the Collector must buffer every span of every in-flight trace in memory until the trace finishes (a buffering window of, say, 10–30 seconds), then evaluate the policy on the assembled trace. That makes tail sampling stateful, memory-hungry, and sensitive to the buffering window — too short and you decide on incomplete traces; too long and you blow the Collector’s memory. It also forces all spans of one trace to reach the same Collector instance, which complicates a horizontally scaled gateway.

The cardinality trap: a metric’s cost is its number of unique label combinations

Before you add a label to any metric, ask: how many distinct values can this field take across your whole user base? If the answer is “one per user” or “one per request,” stop. Here is why.

This is the failure mode that ends careers’ worth of credibility, and it is purely about arithmetic. A metric is not one thing — it is one time series per unique combination of label (dimension) values. The metric http.server.duration{route, status_class, region} with, say, 20 routes × 5 status classes × 4 regions is 400 time series — totally fine. The cost is the product of the cardinalities of every label. The trap is putting a high-cardinality identifier in a label: user_id, request_id, session_id, an email, or a raw URL with IDs baked in (/orders/4821). A user_id label does not add some series — it adds one series per user. Tens of thousands of users is tens of thousands of series; 1.4 million users is 1.4 million series from that one label, and if it multiplies with your existing labels, tens of millions. A Prometheus/AMP backend holds active series in memory, so this OOM-kills it; CloudWatch bills each distinct dimension combination as a separate custom metric (illustratively ~$0.30/metric/month, region-dependent), so a million combinations is a five-figure line item — the Hook’s two bills.

The fix is a discipline, not a feature. Keep metric labels low-cardinality and bounded — values from a small, finite set: the route template (/orders/{id}) not the raw path, status_class (2xx, 5xx) not the exact code if you must bound it, region/az, service.version. High-cardinality identifiers belong in traces and logs, where they are supposed to live — a trace already records user_id and request_id as span attributes for free, and a structured log carries them as fields — and you reach them via the trace_id join, not via a metric label. With EMF specifically: a field in the EMF JSON is only billed as a metric dimension if you list it in the Dimensions array, so log user_id as a plain field (free, queryable) and never put it in Dimensions. The rule a senior internalizes: labels are for things you group and alarm on; identifiers are for things you look one up by — and those go in traces and logs, never in a metric label.

# ANTIPATTERN — explodes to one series per user/request:
attributes:
  - key: user_id      # millions of values -> millions of series
  - key: http.target  # raw "/orders/4821" -> one series per order id

# FIX — bounded labels on the metric; identifiers move to spans/logs:
attributes:
  - key: http.route   # template "/orders/{id}" -> ~tens of series
  - key: status_class # "2xx"/"4xx"/"5xx" -> a handful
  - key: region       # finite set
  # user_id / request_id: keep as SPAN attributes + log fields, not metric labels.
  #   reach them by joining on trace_id, never by adding a metric dimension.