How flame graphs are built from samples, and the production workflows that use them

A senior engineer at a conference asks: “which function runs before this one in the flame graph?” They point to two adjacent frames at the same level. The answer is: neither — the x-axis is alphabetical. If you do not know this, you will waste hours on the wrong hypothesis.

How the flame graph is built from samples

Each sample is a list of function names from leaf (currently executing) to root (program entry). After a profiling window:

1. All samples are collected.
2. Identical stacks collapse into one column — their counts add up, making the column wider.
3. For rendering, all unique stacks are sorted alphabetically by root function — so the same parent groups its children next to each other.
4. For each level, rectangles are emitted with width proportional to the count.

The result reads top-to-bottom: pick a leaf frame (top), check its width, walk down to see what calls it.

The most expensive misread

The x-axis position is alphabetical order of the full stack path — not time, not call order. A frame appearing to the left of another at the same level tells you nothing about which one ran first. It tells you only that its stack path comes earlier alphabetically.

This is the most common mistake engineers make when first reading flame graphs. If you see two wide frames side by side and think “A runs, then B runs,” you are drawing a false conclusion. Both could be called by the same parent at different points; both could be from unrelated code paths; both could be parallelised.

If you need time order, the right tool is a trace view (Gantt-style span timeline). Flame graphs answer “what” is hot; traces answer “when” in the request each step ran.

Profile workflows in production

SLO burn drilldown: SLO alert fires → click link → time range pre-filtered to burn window → CPU + off-CPU flame graphs side by side → identify changed function → blame the deploy. Under 90 seconds from pager to git blame for any incident where the bug ran on CPU.

Deploy regression detection: Capture a profile on both the pre-deploy and post-deploy version under comparable load. Diff them: a differential flame graph colours frames by relative change — red for frames that grew, blue for frames that shrank, white for unchanged. New wide red frames that were absent before the deploy are the regression. Production-grade continuous-profile backends (Pyroscope, Datadog) bake this in: “compare versions” picks two commits or time windows and renders the diff.

Profile-as-data: queries beyond flame graphs:

Profiles are time-series of stack samples — backends increasingly let engineers query them like a database:

• “Top 10 functions by self-CPU across all services for the past hour” → capacity planning.
• “Find all profiles where function X appears in the top 5” → impact assessment before deleting a slow library.
• “Group flame graphs by Kubernetes node” → spot hot nodes.
• “Alert when a new function appears in the top 5 after a deploy” → automated regression detection.