Profile types: CPU, memory, off-CPU, mutex — which one to reach for

A span takes 500 ms. You open the CPU profile — the service used 20 ms of CPU. Where did the other 480 ms go? CPU profiling is blind to waiting. If you stop there, you will never find the answer.

CPU profiles: what they see and what they miss

CPU profiling samples the call stack while the thread is on the CPU — running instructions. A request that spends 20 ms computing and 480 ms waiting on a database query will show 20 ms in the CPU profile and leave 480 ms invisible.

This is the most important constraint in profiling: CPU profiles only see functions that are consuming the processor. Anything the program waits for — I/O, network, locks, scheduling — is off-CPU and invisible to a CPU profiler.

Memory and allocation profiles

Heap profilers sample allocations, not CPU. Go’s heap profile samples one allocation per ~512 KiB (Poisson distributed) and records the stack at each sample. The result is a flame graph where width is allocated bytes, not CPU time. This finds memory hotspots: a function allocating 100 MB/s shows up wide.

Memory leak detection with heap profiles:

1. Take a heap profile.
2. Wait 30-60 minutes.
3. Take another heap profile.
4. Diff them (go tool pprof -base baseline.heap current.heap).
5. Functions whose allocation grew are leaking.

Allocation profiles capture short-lived allocations that GC reclaims; heap profiles snapshot live memory. Both are useful. JVM equivalents: async-profiler with -e alloc, JFR allocation events. Python: tracemalloc, memray.

Off-CPU profiles

Brendan Gregg’s 2013 work on off-CPU analysis identified the gap: CPU profiles miss everything a process waits for. eBPF implementations hook into the kernel scheduler’s switch events. When the scheduler removes a thread from the CPU (it blocks on I/O, sleeps, waits on a lock), the kernel captures the thread’s stack. That stack represents where the wait started. When the thread comes back, elapsed time is attributed to that stack.

The off-CPU flame graph shows wait time the same way a CPU flame graph shows running time. For an I/O-bound service, the off-CPU profile is the only profile that explains anything — the CPU profile just says “service was idle.”

Block and mutex profiles

Block profile (Go: runtime.SetBlockProfileRate): time spent waiting on synchronisation primitives — channels, condition variables, WaitGroups. More focused than off-CPU because it targets language-level synchronisation.

Mutex profile (runtime.SetMutexProfileFraction): lock contention specifically. Reports which code held a lock while others were waiting for it, attributed at unlock time.

Choosing the right profile from the CPU/wall-time ratio

The diagnostic shortcut: look at CPU time vs wall-clock time for the slow request.

• CPU/wall ≈ 100%: computation bottleneck — CPU profile.
• CPU/wall < 30%: the bottleneck is off-CPU — off-CPU / block profile or trace spans.
• Memory growing steadily: heap / allocation profile.
• Threads contending on a lock: mutex profile.

A Java service GC-thrashing is a classic allocation-profile case. The symptom is high heap allocation rate with frequent old-gen GC. The allocation flame graph will show the widest frame as the function allocating at highest rate — often string concatenation in logging code that is not using parameterized formatting.