How flame graphs are built from samples, and the production workflows that use them

How the flame graph is built from samples

Each sample is a list of function names from leaf (currently executing) to root (program entry). After a profiling window:

1. All samples are collected.
2. Identical stacks collapse into one column — their counts add up, making the column wider.
3. For rendering, all unique stacks are sorted alphabetically by root function — so the same parent groups its children next to each other.
4. For each level, rectangles are emitted with width proportional to the count.

Together these four steps explain why the x-axis is alphabetical and not time: the sort in step 3 groups related paths, not sequential events. Without this understanding, every frame you see to the left of another will tempt you into a false “this ran first” conclusion.

The result reads top-to-bottom: pick a leaf frame (top), check its width, walk down to see what calls it.

The most expensive misread

The x-axis position is alphabetical order of the full stack path — not time, not call order. A frame appearing to the left of another at the same level tells you nothing about which one ran first. It tells you only that its stack path comes earlier alphabetically.

This is the most common mistake engineers make when first reading flame graphs. If you see two wide frames side by side and think “A runs, then B runs,” you are drawing a false conclusion. Both could be called by the same parent at different points; both could be from unrelated code paths; both could be parallelised.

If you need time order, the right tool is a trace view (Gantt-style span timeline). Flame graphs answer “what” is hot; traces answer “when” in the request each step ran.

So the read is purely vertical: scan for the widest plateau, because width is sample count and a wide frame is where the CPU actually spent its time — that is your optimisation target. A tall but narrow tower is a deep call chain that is rarely on-CPU; it is a red herring, no matter how dramatic its height looks. The flat top of a wide stack is the leaf actually doing the work, since nothing wider sits above it.

Profile workflows in production

SLO burn drilldown: SLO alert fires → click link → time range pre-filtered to burn window → CPU + off-CPU flame graphs side by side → identify changed function → blame the deploy. Under 90 seconds from pager to git blame for any incident where the bug ran on CPU.

Deploy regression detection: Capture a profile on both the pre-deploy and post-deploy version under comparable load. Diff them: a differential flame graph colours frames by relative change — red for frames that grew, blue for frames that shrank, white for unchanged. New wide red frames that were absent before the deploy are the regression. Production-grade continuous-profile backends (Pyroscope, Datadog) bake this in: “compare versions” picks two commits or time windows and renders the diff.

Profile-as-data: queries beyond flame graphs:

Profiles are time-series of stack samples — backends increasingly let engineers query them like a database:

• “Top 10 functions by self-CPU across all services for the past hour” → capacity planning.
• “Find all profiles where function X appears in the top 5” → impact assessment before deleting a slow library.
• “Group flame graphs by Kubernetes node” → spot hot nodes.
• “Alert when a new function appears in the top 5 after a deploy” → automated regression detection.