Production debugging with no repro: perf, eBPF, flame graphs, and why ptrace stops the world

The production constraint: you cannot stop the world

Every debugger technique from the previous lessons assumes you can halt the process. In production that assumption inverts. The classic interactive debugger attaches via ptrace, and a ptrace attach (PTRACE_ATTACH, what gdb and Delve do) stops every thread of the target until you detach. On a live server taking traffic that is catastrophic: requests queue or time out, health checks fail, the load balancer ejects the instance and dog-piles its siblings, and an autoscaler may kill the “unhealthy” pod you were debugging — destroying the live state you attached to read. So the production toolkit is built on a different principle: observe without stopping. You sample or trace the process as it runs, paying a few percent overhead, and you reconstruct what it is doing from those samples rather than freezing it to inspect.

There is a second reason ptrace is the wrong default in prod: it is single-target and intrusive, while the questions are usually statistical — “where does CPU go”, “what syscall is slow”, “which path allocates” — answered far better by aggregating thousands of cheap samples than by one frozen snapshot.

Sampling vs instrumenting: perf, eBPF, async-profiler

When you pick up a production observability tool, the first decision — sampling or instrumenting — determines whether your overhead is predictable or whether you accidentally crush the service you’re trying to diagnose. Getting this wrong is how a “quick look” turns into an incident of its own.

The low-overhead toolkit splits along sampling versus instrumenting. Sampling interrupts the process at a fixed frequency — perf record -F 99 takes 99 stack samples per second per CPU — and infers where time goes from the distribution; overhead is roughly 1-5% because most cycles run untouched and only the periodic interrupt costs anything. Instrumenting attaches to specific events — a function entry, a syscall, a tracepoint — and runs on every occurrence; cheap per-event but costly if the event is hot. perf, the kernel’s sampler, profiles CPU with hardware performance counters. eBPF (driven by bpftrace) runs a verified, sandboxed program in the kernel on chosen events, aggregating in-kernel so even per-event tracing stays cheap — bpftrace -e 'tracepoint:syscalls:sys_enter_read { @[comm] = count(); }' counts reads by process with negligible cost, no process stop, no kernel module. For managed runtimes, async-profiler samples the JVM (or similar) using a signal-based or perf-events mechanism that, unlike naive JVMTI stack-walking, does not suffer safepoint bias — it can sample threads at arbitrary points, not just where the runtime parks them, so the flame graph reflects real hot paths rather than safepoint locations.

USDT probes and flame graphs: locating the path live

Two pieces turn samples into an answer. Flame graphs aggregate thousands of stack samples into a single picture: each box is a function, width is the share of samples (time), stacked by call depth. A wide box near the top is where CPU actually goes — the regex-compile plateau from the Hook reads off in one glance, no stepping required. USDT probes (userland statically-defined tracing) are zero-overhead-when-disabled markers compiled into a binary (databases, language runtimes, your own service) that eBPF/perf can attach to at named, semantically meaningful points — “transaction commit”, “query start” — giving you stable instrumentation without modifying or stopping the process. Together they let you answer “what is this wedged process doing right now” on a live, untouched service: sample it, fold into a flame graph, read the hot path, and confirm with a USDT or kernel tracepoint that the suspect path is the one firing. The whole investigation costs a few percent CPU and zero downtime, which is the only kind of debugging a production SLA allows.