Flame graphs: reading the picture that shows where time goes

A trace says “1.2 seconds in inventory.” You have logs, metrics, dashboards — but none of them tell you which function inside inventory ate the time. Profiling answers that question in 60 seconds without a debugger or a guess.

What a profiler does

A profiler interrupts your program 100 times per second and captures the current call stack — the chain of functions from main down to whatever is running right now. After 30 seconds it has 3,000 snapshots. The function that appears most often across those snapshots is the one consuming the most CPU.

Flame graphs visualise this: stacks are sorted alphabetically along the x-axis, width is proportional to sample count, depth on the y-axis goes from the entry point at the bottom to the leaf function at the top. The widest frame at any level is the busiest path.

The stadium metaphor

Imagine a stadium with 100,000 people doing different things. A helicopter flies overhead 100 times per second and photographs who is doing what. After a minute you have 6,000 photos. Count which activity appears most across the photos — that is where the crowd’s “CPU time” goes.

The flame graph is the bar chart of those counts, with callers stacked beneath callees. Wide bars = popular activities. The helicopter is the profiler; the photos are stack samples; the chart is the flame graph.

Reading a flame graph in practice

Bea is on-call. Inventory service, p99 = 1.5 s. She opens the continuous-profile dashboard, filters by trace-id, and sees a flame graph with one massive 1.1-second-wide block: json.Marshal inside serializeResponse. The fix is obvious: cache the marshalled response or pre-encode at write-time. Without profiling, the team would have guessed — DB? Cache? Network? With the flame graph there is no guessing.

How to capture a CPU profile with pprof

A Node API has a p99 jump. Tracing finds a slow span. The continuous-profile dashboard, filtered by trace-id, shows a flame graph dominated by a regex compile in a handler. A library upgrade introduced an O(n²) regex; fix is to precompile it outside the handler. Detection: 60 seconds.

For Go services, pprof is built-in:

// 1. Expose pprof handlers (registers /debug/pprof/* routes)
import _ "net/http/pprof"

// Start the debug server
go func() {
  http.ListenAndServe("localhost:6060", nil)
}()

// 2. Capture a 30-second CPU profile under load:
//    go tool pprof -http=:9090 \
//      http://localhost:6060/debug/pprof/profile?seconds=30
//
// 3. The flame graph view opens at :9090.
//    Widest top-level leaf = hot path.

You must run the profile under representative load — on an idle system, almost everything in the samples is the runtime’s idle loop, useless for finding hot paths.