What the three signals are: logs, metrics, and traces

The three signals

Cindy Sridharan’s 2018 O’Reilly book Distributed Systems Observability crystallised the vocabulary: production systems emit three telemetry shapes.

• Metrics — numeric measurements aggregated over time with a small fixed set of labels. Pre-aggregated, cheap to query across years of history, blind to any dimension you did not label in advance.
• Logs — event records (timestamp + payload). Preserve every field you wrote, at the cost of ingestion bytes and query scan time.
• Traces — the causal chain of one request across services. Preserve causality and timing per span, at the cost of sampling — storing every trace would multiply storage by an order of magnitude over the original traffic.

Together, each signal fills the blind spot of the other two: metrics see the trend but not the record, logs see the record but not the path, traces see the path but not the population. The three are not redundant. Each preserves something different and pays a different cost. A team with only one signal cannot triage end-to-end.

The kitchen metaphor

A restaurant kitchen maps the three signals clearly. Metrics are wall dials — covers per hour, average ticket time, error rate. Cheap to read, always-on, but they only show what you labelled in advance. Logs are order tickets on the rail — every dish, every modifier, every table. Bulky, but when a customer complains you go to their ticket. Traces are the story of one specific order: prep at station A, sauté at B, plate-up, runner, table. Sampled — tracing every dish would bury you — but a cold dish lights the whole path.

The triage walk

Bea · Browser gets paged: checkout error rate jumped to 3%. She opens metrics — sees the spike at 14:02 UTC. The metric says “yes, errors are real” but not “why.” Sven · Origin server pulls logs filtered to status=5xx — sees a stack trace pointing at a payment-gateway timeout. Logs gave the cause. To know whether the request retried or wrote stale state, they pick a trace_id from the log line, open the span tree, and see the gateway timing out after 30 s with no retry. Three signals, three jobs, seconds each.

The rule: metrics are the alarm, logs are the diagnostic, traces are the deep dive.