Observability
Observability capstone: multiple-choice synthesis
Crux Cross-track synthesis MCQs spanning the whole observability chapter — funnel order, the trace-id join, tail sampling, cardinality cost, MTTD vs MTTR, and ROI.
Your altitude — climbing toward senior
ZeroJuniorMiddleSenior
You are at senior altitude — in orbitSix questions that force you to wire the whole track together — three pillars, OTel, RED/USE, SLO budgets, trace propagation, profiling. Each one is a decision you make mid-incident or mid-design-review, not a definition to recite.
Goal
Confirm you can compose the chapter into one system: the funnel that drives MTTR, the trace-id that joins four signals, the sampling and cardinality levers that hold the bill, and the arithmetic that justifies the spend.
Quiz
Completed
An on-call is paged 'checkout SLO burn 14x'. RED shows p99 Duration up, Rate and Errors flat. What is the correct next move in the funnel, and why that one?
Heads-up A profile needs a target — a service and ideally a trace-id — before it is useful. Jumping to the profile before the trace skips the step that tells you which service and span to profile.
Heads-up Errors are flat; this is a latency incident, not an error incident. Logs are the funnel's last resort for specific event lookups, not the second step.
Heads-up You have not yet localised the cause. Rolling back blind may revert an unrelated change and still leave the latency; the funnel localises first, then resolves.
Quiz
Completed
A request slows down. RED Duration shows it, the trace shows inventory.lookup owns 1.3s, the profile shows json.Marshal ate the CPU, and the logs hold the error detail. What single property makes all four signals answer one query about THIS request?
Heads-up Timestamp + service name is exactly the manual correlation the trace-id replaces. Clock skew and concurrent requests make timestamp-only joins ambiguous; the trace-id is an exact key.
Heads-up Co-locating signals in one backend helps UX but does not create the join. Tempo, Loki, Prometheus and Pyroscope are separate stores yet still join — because each row carries the same trace-id.
Heads-up Semantic conventions make cross-service dashboards consistent, but the per-request join across signal TYPES is the trace-id specifically — conventions are about attribute keys, not request identity.
Quiz
Completed
A team head-samples traces at 5% at the SDK to cut cost. During an incident they cannot find traces for the failing requests. What went wrong, and what is the fix?
Heads-up Head sampling is probabilistic at trace start and has no knowledge of the eventual outcome — it cannot preferentially keep errors it has not seen yet. This is the intrinsic limitation, not a misconfiguration.
Heads-up Traces are precisely the per-request causal signal you want in an incident. The problem is the sampling strategy, not the signal.
Heads-up 100% head sampling defeats the cost goal — you store every baseline trace forever. Tail sampling keeps 100% of what matters and drops 90%+ of the boring baseline, getting both cost and coverage.
Quiz
Completed
The observability bill doubled in a week with no traffic change. One team added user_id as a metric label and started logging full request bodies at INFO. Which two distinct failure modes is this, and which signal does each hit?
Heads-up Neither touches traces. user_id hits metric cardinality; request bodies hit log content and compliance. Tail sampling addresses neither.
Heads-up Tiering controls how long data lives, not its width or its PII content. An unbounded label still explodes series on day one; tiering does not stop the leak.
Heads-up Encryption at rest does not stop an authorised query (or an attacker with access) from reading the PII. Both risks are real and documented — Datadog 2021 (cardinality) and Slack 2022 (PII in logs).
Quiz
Completed
Two teams share identical tooling. Team A detects an outage in 30s and resolves in 20min; Team B detects in 8min and resolves in 12min. For the same severity, which team's users suffer less total impact, and what does that say about where to invest?
Heads-up MTTR alone ignores the failure window before detection. B's users absorbed 8 minutes of unflagged failure before the clock even started; total impact is roughly MTTD + MTTR per affected request.
Heads-up Impact is not symmetric across the two phases — during the detection delay the failure is fully live and unmitigated. A 30s MTTD with a 20min MTTR still exposed users for far less of the dangerous window than 8min undetected.
Heads-up The tooling is identical by stipulation; the entire difference is the MTTD vs MTTR shape, which is a process and alerting question, not a vendor question.
Quiz
Completed
A CFO asks why observability costs $150k/year. The org runs ~30 incidents/year, each now resolved ~25 min faster than the pre-instrumentation baseline, at roughly $5k/min of revenue exposure. What is the strongest answer?
Heads-up Industry-standard is not a business case. The defensible answer is measurable avoided-outage cost derived from your own incident history.
Heads-up True but not a quantified justification — it is the kind of answer that gets the budget cut in a downturn. Tie the spend to revenue protected or risk reduced.
Heads-up SLAs are downstream of observability; the business argument is the avoided-cost arithmetic, and it is computable from MTTR logs, incident counts, and revenue rate.
Recap
The chapter composes into one system. The funnel (SLO burn → RED → trace → profile → git blame) drives diagnosis, and each step is forced by the previous one. The trace-id propagated by W3C traceparent is what makes four signals answer one per-request query. Cost discipline is three independent levers — cardinality limits, tail sampling, tiered retention — each hitting a different signal. MTTD usually outweighs MTTR for user impact because the detection window is fully unmitigated. And the spend is justified by arithmetic: avoided-outage cost from real MTTR deltas, typically 10–30x. Every answer here is a drill-down from those through-lines.