Observability
Profiling: multiple-choice review
Crux Multiple-choice synthesis across the profiling unit — sampling economics, profile-type selection, flame-graph reading, eBPF symbolization, and production discipline.
Your altitude — climbing toward senior
ZeroJuniorMiddleSenior
You are at senior altitude — in orbitSix questions that cut across the whole unit. Each one mirrors a decision you make in a real latency incident — not a definition to recite, but which profile to pull, how to read it, and what its numbers actually mean under production load.
Goal
Confirm you can connect sampling economics, profile-type selection, flame-graph reading, eBPF symbolization, and production discipline — the synthesis the individual lessons built toward.
Quiz
Completed
A request takes 500 ms wall-clock but the CPU profile attributes only 20 ms to it. Where did the other 480 ms go, and which profile reveals it?
Heads-up A higher rate gathers more samples of running code, but a descheduled thread consumes zero CPU and is never on the CPU to be sampled at any rate. The time is off-CPU by definition.
Heads-up A heap profile shows allocated bytes per stack, not wait time. The general answer for a low-CPU/high-wall request is off-CPU waiting, surfaced by an off-CPU or block profile.
Heads-up Nothing was dropped. The CPU profile correctly reports 20 ms of compute; the rest is off-CPU time a CPU profiler is designed not to see.
Quiz
Completed
A service makes 5 million function calls/second. You want always-on production profiling. Why is a sampling profiler the only viable choice over instrumentation?
Heads-up Instrumentation captures exact call counts and timings per function — more data, not less. The disqualifier is cost: it scales with call frequency, not the data it produces.
Heads-up Instrumentation is exact; sampling is statistical with a confidence interval. Sampling wins on bounded overhead, not accuracy — its accuracy is merely sufficient to find hotspots.
Heads-up Both need representative load — an idle system shows only the runtime idle loop. The reason sampling wins in production is its bounded, call-rate-independent overhead.
Quiz
Completed
An engineer points at two wide frames side by side at the same level of a flame graph and concludes the left one runs before the right one. What is wrong?
Heads-up This is the single most common flame-graph misread. The flame graph sorts stacks alphabetically by root to keep a parent's children adjacent; the x-axis is never time.
Heads-up They may or may not share a parent. Adjacency comes from lexicographic sorting of the full stack string, and width is sample count — neither encodes when or how often they alternated.
Heads-up No standard flame graph puts time on the x-axis. A wall-clock view of one request is a flame chart / trace timeline, a different visualisation.
Quiz
Completed
An eBPF profiler shows clean stacks for Go and Rust services but many '[unknown]' frames for a Python service on the same node. Why?
Heads-up Speed is irrelevant to symbolization. The kernel captures the stack fine; it just cannot map interpreter-internal addresses to Python names without CPython-specific unwinding.
Heads-up [unknown] frames for interpreted runtimes are expected without explicit support. The fix is a Python-aware profiler (py-spy) or a CPython-supporting eBPF mode, not replacing the agent.
Heads-up CPython is a bytecode interpreter, not a JIT. JIT runtimes (JVM, V8) have a different symbolization problem solved via perf maps or AsyncGetCallTrace.
Quiz
Completed
A continuous profiler is advertised at 2-5% overhead, but you measure 12% in production. What is the most likely cause to check first?
Heads-up The 2-5% figure is real and verifiable on typical workloads. A spike above it almost always indicates misconfiguration or a deep-stack/JIT-symbolization cost — check config before swapping tools.
Heads-up Continuous profiling at 2-5% is industry-standard always-on. The premise of the unit is that it is cheap enough to leave on; a 12% reading is an anomaly to diagnose.
Heads-up Sampling cost is bounded by sample rate, not call rate, so it does not grow with traffic. A rise to 12% points at rate misconfig, synchronous symbol work, or deep stacks.
Quiz
Completed
During an incident review someone proposes attaching the production profile to the vendor support ticket. Why should a senior engineer object?
Heads-up This is exactly the misconception the unit warns about. A profile is closer to debugger output than to a metric; function names and call patterns can reverse-engineer business logic.
Heads-up CPU profiles expose function names and call patterns already. Allocation profiles add possible argument values on top — but the CPU profile alone warrants access control.
Heads-up Compressed profiles are tiny (~50-500 KB). The concern is information exposure, not file size.
Recap
The unit’s through-line is one diagnostic loop: pick the profile that answers your question (CPU for compute, off-CPU/block/mutex for waiting, heap/allocation for memory), read the flame graph correctly (width is sample share, x-axis is alphabetical not time), and trust the numbers only when you know their limits (sampling is bounded but blind to sub-interval hot paths, eBPF needs language-aware symbolization, 2-5% overhead is real but config-sensitive). And because a profile exposes code internals, you treat it as a security-sensitive artefact, not a freely shareable metric.