Continuous profiling: always-on flame graphs with eBPF and trace-id correlation

An SLO burns at 2:14 AM. The pager wakes the on-call. Traditional profiling requires them to SSH in, reproduce the issue under load, capture a profile, and parse it — at 2:14 AM, under pressure. Continuous profiling already has the flame graph for 2:14 AM waiting in a dashboard.

Traditional vs continuous profiling

Traditional (on-demand) profiling: SSH in, run perf record or hit /debug/pprof/profile, gather data, analyse, leave. Cost: only during capture. Limitation: requires you to be present and the issue to be actively occurring. For intermittent issues or incidents that self-resolve, you lose the evidence.

Continuous profiling: an agent runs on every host or container, sampling 100 times per second continuously, batching and shipping compressed profiles every 10-15 seconds to a backend. The backend stores them indexed by service, host, and time range. Storage: ~50-200 MB/day per service, ~1.5-6 GB/month. Overhead: 2-5% CPU. The critical win: when an SLO burns, the profile of the burning minutes is already saved.

eBPF: language-agnostic profiling

Traditional language profilers (Go pprof, JFR, async-profiler, py-spy) require language-specific support — the runtime must expose stack walking APIs. For Python, Ruby, and older PHP interpreters, this requires hooks the runtime team must provide.

eBPF profilers (Pyroscope eBPF mode, Parca) read stacks from the kernel side: the kernel’s perf_event_open syscall plus a BPF-attached probe captures user-space stacks at sample time. This means:

• Works for any language, any binary, with no application code change.
• One agent covers all services on the host — Go, Java, Node, Python.
• Catches third-party library overhead that language-specific profilers might miss.

The catch: symbol resolution. The kernel sees memory addresses; the profiler maps them back to function names using debug info (DWARF, BTF, JIT-emitted symbol files for V8 or JVM). Most production eBPF profilers handle this; occasional [unknown] frames appear when DWARF is stripped or JIT code is too volatile.

Cross-language profiler coverage

Trace-id correlation: from slow span to flame graph in 30 seconds

Each profile sample can carry the trace-id of the request being processed at the moment of sampling — stored in thread-local context. When a slow trace appears in the trace view, the matching profile samples (only those carrying that trace-id) can be filtered out and rendered as a flame graph for that specific request.

This is the bridge between “where did time go in the request” (trace span) and “what code ate the CPU” (profile). The workflow:

1. SLO alert fires — p99 latency over budget.
2. Open trace view — find slow spans, note trace-id.
3. Open profile view filtered by trace-id — flame graph for that exact request appears.
4. Widest frame is the function to fix.
5. Done in under 60 seconds.

OpenTelemetry’s profile signal (stabilising in 2025-2026) standardises this linkage. Production-grade observability platforms (Datadog, Grafana with Pyroscope, Honeycomb) ship this drilldown out of the box.

Profile storage economics

Pyroscope 2.0 (released April 2026) cut storage 3x by deduplicating symbols across profiles from the same binary — function names and source paths are shared in a common symbol table instead of repeated in every profile.

Retention strategy: 7 days full-fidelity for active debugging, 30 days downsampled (one profile per 5 minutes), 90 days for long-term trend analysis. Budget-conscious teams cap at 14 days fine + 60 days coarse.