pprof end to end: CPU, heap, goroutine and mutex profiles from a live service to the leak

Instrumenting production: net/http/pprof done right

import _ "net/http/pprof" registers handlers under /debug/pprof/ — on http.DefaultServeMux, which is the part that bites. If your main server uses the default mux, your profiling endpoints are now public: an internet-facing /debug/pprof/ leaks symbol names, source paths, and lets anyone trigger 30-second CPU captures. The production pattern is a separate internal listener:

import (
	"net/http"
	_ "net/http/pprof" // registers on http.DefaultServeMux
	"runtime"
)

func main() {
	runtime.SetMutexProfileFraction(100) // sample 1/100 mutex contention events
	runtime.SetBlockProfileRate(10_000)  // sample blocking ≥10 µs (ns units)

	go func() {
		// internal port only: cluster network / localhost, never the LB
		http.ListenAndServe("127.0.0.1:6060", nil) // serves DefaultServeMux
	}()
	// main API server on its own mux, own port …
}

// Capture from a workstation (port-forward in k8s):
//   go tool pprof -http=:8080 "localhost:6060/debug/pprof/profile?seconds=30"
//   go tool pprof -http=:8080 "localhost:6060/debug/pprof/heap"
//   curl -o now.pb localhost:6060/debug/pprof/heap   # snapshot for later diffing

Overhead is the perennial objection, and the honest numbers defuse it: the CPU profiler samples at 100 Hz and typically costs low single-digit percent while capturing — and nothing when idle. The heap profile is sampled at allocation time (default: one sample per ~512 KB allocated) and is always on; reading it is nearly free. Mutex and block profiles are off by default (fraction/rate = 0) and cost only what your sampling rate asks for. Leaving pprof reachable on an internal port is standard practice at every Go shop that has been paged twice.

The profile menu: which one answers which question

• CPU (/debug/pprof/profile?seconds=30): where do cycles go? The runtime interrupts each executing thread 100 times a second and records the stack; the profile is a statistical census of 3,000 samples over 30 s. It sees on-CPU time only — a service that is slow because it waits (locks, I/O, GC assist) can show an almost empty CPU profile.
• Heap (/debug/pprof/heap): two views of the same data, and choosing wrong wastes an afternoon. inuse_space — bytes currently live, attributed to their allocation site: the question “what is filling memory?” — leaks, caches, retention. alloc_space — cumulative bytes ever allocated: the question “what is generating GC pressure?” — churn, the previous lessons’ fmt boxing. A site can dominate alloc_space while contributing zero inuse (allocate-and-drop churn), and vice versa (one giant retained slice).
• Goroutine (/debug/pprof/goroutine): every goroutine grouped by identical stack — the leak census from the concurrency unit.
• Mutex and block: where contended locks and channel/select waits burn wall time. These answer the empty-CPU-profile riddle: the service that is slow while idle-looking. Both require opt-in sampling rates (see code above) — a mutex profile that shows nothing usually means it was never enabled, not that contention is absent.

Together these four profiles form a decision tree: if CPU is hot, read the CPU profile; if memory grows, compare inuse_space snapshots; if latency is high but CPU looks idle, enable and read mutex/block. Without the right profile, you’re guessing at the wrong layer.

Reading flamegraphs: width is truth, height is noise

go tool pprof -http=:8080 profile.pb serves the interactive UI; the flamegraph view is where most reading happens. The rules: width = inclusive cost (that function plus everything it called), so the widest boxes are where the money is; height is just call depth — a tall narrow spike is a deep stack costing little, and chasing it is the classic rookie detour. When you open a flamegraph for the first time, resist the instinct to follow the tallest tower — find the widest plateau instead. Look for wide plateaus at the top edge: a box with no children burning width is self time — actual loops, memcpy, syntax-level work — and is where optimization lands. Two systematic moves beat staring: the top view sorted by flat (self) versus cum (inclusive) tells you whether a function is expensive itself or merely presides over expensive children; and “focus” (click, or -focus=regex) re-renders the graph restricted to one subtree, which turns a 60-thread service profile into a readable story. Recurring Go-specific shapes: a wide runtime.gcBgMarkWorker tower means GC pressure (go fix allocations, not the GC); wide runtime.mallocgc under your own functions is the same message delivered locally; runtime.futex/runtime.semacquire plateaus point at lock contention — confirm with the mutex profile, not the CPU one.

The leak workflow: diff, then continuous profiling

The Hook’s two-snapshot move is the canonical leak workflow and deserves to be mechanical reflex: capture a baseline (curl -o base.pb …/heap), wait for growth (an hour, a day — long enough for signal to dominate noise), capture again, and run go tool pprof -base base.pb current.pb. The diff subtracts the stable heap — pools, caches, steady-state buffers — and what remains is growth attributed to allocation site. From there the question is always “who retains this?”, and the answer is usually one of: an append-only package-level structure (the Hook), a map used as cache with no eviction, goroutine leaks holding captured buffers (cross-check the goroutine profile), or subtle retention via subslicing a large array. Continuous profiling is this workflow productized: an agent (Grafana Pyroscope, Parca, cloud vendors’ profilers) captures profiles every few minutes, stores them as a time series, and lets you diff across a deploy boundary or ask “what changed at 14:32?” The overhead conversation is the same as above — sampling profilers at low frequency cost ~1–2% — and the payoff is that the profile from before the incident exists, which an on-demand curl can never give you.