Join keys and exemplars: making the three signals compose

A metric shows checkout p99 spiking. You open the log search — but your metric uses route="/checkout", the log uses http_path="/checkout", and the trace uses http.route="/checkout". Three different field names, three disconnected searches. The pivot that should take 10 seconds takes 20 minutes.

The join key problem

Three signals on three different backends are useless if you cannot navigate between them. The bridge is a small set of shared attribute keys that appear identically in metrics, logs, and traces. When they diverge — even subtly — every dashboard needs ad-hoc query translation and every on-call run-book is fragile.

The standard set, formalised in OpenTelemetry Semantic Conventions:

• service.name — which service emitted this
• trace_id + span_id — which request, which step
• http.route — the templated route (e.g. /orders/{id}, not the full URL /orders/42)
• http.request.method, http.response.status_code
• db.system, db.operation, db.namespace
• messaging.system, messaging.destination.name

When metrics carry the same service.name and http.route as log lines, and log lines carry the trace_id of the request that emitted them, and traces carry the same http.route attribute as the metric — that is when “click from metric spike to log line to trace” works in under 30 seconds. Without those join keys you have three disconnected dashboards and no narrative.

Exemplars: the metric-to-trace bridge

An exemplar is a sampled trace_id attached to a histogram bucket observation. When http_request_duration_seconds records a 1.5 s observation in the bucket above 1 s, the histogram client optionally attaches one trace_id that produced that observation. Grafana renders these as dots on the histogram heatmap; clicking a dot opens the corresponding trace.

Exemplars (standardised in Prometheus 2.32, early 2022, now standard in OTel exporters) eliminate the gap that defined the pre-exemplar era: “metrics say something is slow but I have no example request to drill into.” The pivot is:

1. Metric histogram spike → click exemplar dot
2. Exemplar opens the trace for the specific slow request
3. Trace span shows db.query took 1.3 s
4. Log filtered to trace_id=<same> shows the full query text

No manual timestamp correlation. No searching across dashboards. One click chain.

OpenTelemetry: the integration layer

The OpenTelemetry project (CNCF, born 2019 from a merger of OpenCensus and OpenTracing) standardises three things:

• API — the surface engineers write code against: tracer.start_span(), meter.create_counter().
• SDK — the per-language implementation that builds OTLP messages.
• OTLP — OpenTelemetry Protocol: binary protobuf over gRPC or HTTP, roughly 50–70% smaller than JSON over HTTP.

The pitch: instrument once, route anywhere. Application code calls the OTel API, the SDK builds OTLP messages, the OTel Collector forwards them to any backend — Honeycomb, Datadog, Jaeger, ClickHouse, Loki, Prometheus — anything that speaks OTLP or has a contrib exporter. Without OTel Semantic Conventions, every team invents their own field names and joins fail at integration time.