Observability
Three pillars: multiple-choice review
Crux Multiple-choice synthesis across the three-pillars unit — signal selection, cardinality, log volume, sampling, join keys, and the 2.0 cost shift.
Your altitude — climbing toward senior
ZeroJuniorMiddleSenior
You are at senior altitude — in orbitSix questions that cut across the whole unit. Each one mirrors a decision you make mid-incident or at instrumentation time — not a definition to recite, but a cost tradeoff to weigh.
Goal
Confirm you can pick the right signal for a question, reason about its cost axis, and connect the three pillars through join keys — the synthesis the individual lessons built toward.
Quiz
Completed
A product manager wants per-customer error rates broken down by individual customer_id across 200k active customers, queryable for the last 30 days. In a 1.0 stack, which signal carries this data, and why not the others?
Heads-up customer_id is unbounded; 200k values multiply the series count by 200k and OOM the TSDB. Per-customer identity belongs in logs or traces, never in a metric label.
Heads-up A Summary computes percentiles per replica and still needs a label per customer to separate them — the cardinality problem is unchanged, and summaries cannot aggregate across replicas.
Heads-up It is entirely possible in logs and traces, which preserve every field. Only the metrics pillar cannot afford unbounded per-customer labels.
Quiz
Completed
Two services share a heavy endpoint. Service A holds a stable label set (route × method × status_class = 400 series) but logs every request. Service B has tiny log volume but added user_id to one counter. Which threatens its observability backend first, and on which axis?
Heads-up Log volume drives a linear ingest bill and query latency, not a hard memory cliff. It is a cost problem, not an OOM-in-minutes problem like a cardinality bomb.
Heads-up They are different cost axes: cardinality is series-count memory in the TSDB (a cliff); log volume is ingestion bytes and scan time (linear). Conflating them is the core confusion the unit dismantles.
Heads-up Auto-sharding helps log ingest but does not save a Prometheus head block from millions of series; an unbounded label still OOMs the TSDB.
Quiz
Completed
A team runs tail-based sampling at the collector keeping 100% of errors and 2% of successes. Their trace bill barely drops versus 100% retention, even though stored volume fell sharply. Why?
Heads-up It works as designed. Storage volume drops, but the collector must still receive and buffer every span to make a context-aware decision — that cost is unavoidable in tail sampling.
Heads-up 100% head sampling saves nothing; head sampling saves by dropping at trace start. The point of tail sampling is keeping rare errors and slow tails that head sampling would randomly miss.
Heads-up Errors are usually a small fraction of traffic. The dominant remaining cost here is collector buffering of all spans, not the kept error traces.
Quiz
Completed
Metrics use route='/checkout', logs use http_path='/checkout', and traces use http.route='/checkout'. The data is all correct and flowing. What actually breaks, and what is the fix?
Heads-up Automated pivots key on the field NAME, not just the value. Divergent names mean every dashboard and run-book carries fragile ad-hoc translation. Naming consistency is the entire join-key thesis.
Heads-up The collector forwards regardless of attribute naming. The failure is operational friction during triage, not a broken pipeline.
Heads-up Cardinality is about label-VALUE counts within one metric, not whether three backends agree on a field name. Naming divergence is a join problem, not a cardinality problem.
Quiz
Completed
A 40-service team spends ~2 engineer-weeks per quarter policing cardinality budgets and ~3 hours per incident on cross-pillar pivots. They also keep 5 years of low-cardinality SLO metrics for compliance. What is the senior recommendation?
Heads-up 2.0 wins for incident response and high-cardinality queries, but raw wide events over 5 years cost more than pre-aggregated metric rollups. 'Always better' is the misconception lesson 6 explicitly refutes.
Heads-up Lowest infra cost ignores the hidden engineering cost: 2 weeks/quarter on label hygiene plus 3 hours/incident on pivots is the real bill driving the 2.0 case.
Heads-up Reaching feature parity with a SaaS wide-event backend takes 1–2 engineer-years; that is only justified for very large orgs with a dedicated observability team, not a 40-service shop.
Quiz
Completed
During an incident the on-call finds the trace they need was never sampled — the SDK uses ParentBased(TraceIdRatioBased) but the trace_id is derived from a hash of the request path. What is the root cause, and what is the durable fix?
Heads-up ParentBased explicitly honours the incoming sampled flag — that is its job. The defect is the non-random trace_id seed, flagged in the Elastic 2024 post-mortem.
Heads-up Head sampling is the cheap baseline and is fine when the trace_id is truly random. The fix is a random seed plus tail policies for the interesting cases, not abolishing head sampling.
Heads-up 100% sampling removes the cost savings that justified sampling and still does not fix the correlation bug. A random seed plus tail policies keeps cost bounded and never loses error traces.
Recap
The through-line of the unit is one decision: the question picks the signal at its cheapest cost axis. Metrics are cheap-to-query but die on cardinality; logs preserve everything but pay in ingestion bytes; traces preserve causality but must sample (and tail sampling shifts cost from storage to collector). Join keys — service.name, trace_id, http.route via OpenTelemetry Semantic Conventions — are what make the three compose into one navigable surface. The 2.0 wide-event model collapses all three when the engineering cost of cardinality discipline and cross-pillar pivots exceeds the bill of a unified backend — an economic call, not an ideological one.