Linux perf, eBPF internals, PGO, and the limits of sampling

A flame graph shows 8% of total fleet CPU in a third-party profiler’s own stack-walking code. The profiler is profiling itself. This is not a bug — it is a symptom of deep stacks and misconfigured symbol resolution, two of the senior-level edge cases that production profiling routinely surfaces.

Linux perf: the kernel foundation

Linux’s perf subsystem is the foundation under most profilers. The kernel exposes perf_event_open(2), which lets user space request sampling on CPU cycles, hardware events, cache misses, or software events. When the configured number of events fires, the kernel captures the current PID, the user-space stack pointer, and writes them to a ring buffer. User space reads the buffer and reconstructs stacks.

Kernel-side cost: ~1-5 microseconds per sample including the stack walk. Modern Linux (5.x+) supports BPF programs attached to perf events, letting user space customise what is captured — filter by PID, attach metadata, do statistical processing in-kernel. This is the engine under Pyroscope eBPF, Parca, BCC’s profile.bt, and any eBPF-based continuous profiler.

The security implication: perf_event_open requires capabilities (CAP_PERFMON in newer kernels, or root historically). Production deployments use kernel-side cgroup permissions to restrict what each container can observe.

eBPF and JIT stack-unwinding nuances

The kernel can walk user-space stacks easily for compiled binaries (Go, Rust, C++) — frame pointers or DWARF unwind info on the stack. For JIT runtimes (V8, JVM, .NET CLR), the stack contains JIT-compiled code addresses that do not correspond to stable symbols.

Profilers handle this with perf maps: JITs emit /tmp/perf-PID.map files mapping address ranges to function names, which the profiler ingests. JVM async-profiler bypasses this by hooking into AsyncGetCallTrace — a JVM API that returns Java stack frames directly. V8 supports --perf-prof / --perf-basic-prof flags.

The challenges:

• Maps can lag behind recompilations — hot functions get re-JIT’d and the old map entry is stale.
• Stack unwinding can fail through certain JIT frames, leaving gaps.
• Production profilers handle 90-95% of stacks accurately; the rest show as [unknown].

Mysterious [unknown] gaps in a flame graph almost always indicate a JIT-frame unwinding issue, not a bug in the application code.

Off-CPU profiling: the scheduler hook

Brendan Gregg’s 2013 work on off-CPU analysis identified the gap that CPU profiles leave. eBPF made off-CPU profiling cheap by hooking the kernel scheduler’s switch events (sched_switch tracepoint). When the scheduler removes a thread from CPU (it blocks on I/O, sleeps, waits on a lock), the kernel’s BPF program captures the thread’s stack and records the timestamp. When the thread comes back on CPU, elapsed time is attributed to the captured stack.

Production usage: off-CPU profiling is essential for I/O-bound services (databases, network gateways) where CPU profiles show ~5% of time and the real work happens waiting. Cost: comparable to CPU profiling (~2-5%) when sampling; unsampled (every scheduler switch) can be expensive on highly-switching workloads. Production deployments cap to ~99 Hz for off-CPU as well.

Profile-guided optimisation (PGO)

Profilers do not just diagnose — they feed back into compilation. PGO workflow:

1. Compile a binary with instrumentation (or use sampling profiles from CPU profiler).
2. Run the binary under representative load to collect a profile.
3. Recompile with the profile as compiler input.

The compiler now knows which branches are hot vs cold, which functions to inline aggressively, how to lay out basic blocks for CPU cache locality. Real-world PGO improvements: Go 1.21+ PGO gives 2-7% speedup on typical workloads (Go’s CPU pprof profile is directly consumable as PGO input). Chrome saw 10%+ from PGO + LTO. PostgreSQL benchmarks 5-15%.

Continuous profiling backends increasingly support exporting profiles in PGO-compatible formats, enabling automated “profile → recompile → validate” pipelines in CI.

Sampling biases and blind spots

Sampling has limits that senior engineers must know:

Short hot paths: functions whose entire runtime is shorter than the sample interval (10 ms at 100 Hz) can be missed. A function called 1 million times for 5 microseconds each (5 s total CPU) might never be sampled — each call ends before the timer fires. Mitigation: longer profiling windows, higher sample rates, or instrumentation for that specific path.

NUMA / multi-CPU asymmetry: profilers usually sample uniformly across CPUs. If work is unevenly distributed (common on NUMA systems or with thread affinity set wrong), the hot CPU’s bottleneck dominates and the profile looks skewed. Use per-CPU profile views to see imbalance.

Statistical accuracy at low sample counts: at 100 Hz for 10 seconds you have 1,000 samples. A function using 1% CPU appears in ~10 samples — enough to notice, insufficient for precise estimates. Standard error is ~30% for functions under 10% CPU. For sub-1% regressions, a longer capture (60-300 s) or differential profile is required.