Benchmarking real services: percentiles, coordinated omission, and the optimization loop

Microbenchmarks: precise answers to small questions

testing.B is excellent at what it actually measures: the cost of a code path in a warm cache, with no neighbors, no GC pressure from the rest of the service, and no network. That makes it the right tool for comparing two implementations of the same function — and the wrong tool for predicting service latency. Even within its scope there are two classic self-deceptions. First, dead-code elimination: if the benchmarked result is unused, the compiler may delete the work; keep results alive (assign to a package-level sink, or use the modern b.Loop() pattern which keeps its body’s results live). Second, missing allocation truth: always run with b.ReportAllocs() — a function that “got 10% faster” while doubling allocs/op is usually a regression in service context, where allocations are the GC tax from previous lessons.

var sink int // package-level: defeats dead-code elimination

func BenchmarkParsePrice(b *testing.B) {
	data := loadFixture()
	b.ReportAllocs()
	for b.Loop() {           // Go 1.24+; replaces for i := 0; i < b.N; i++
		v, _ := ParsePrice(data)
		sink = v
	}
}

// $ go test -bench=ParsePrice -count=10 > old.txt
// … apply the change …
// $ go test -bench=ParsePrice -count=10 > new.txt
// $ benchstat old.txt new.txt
//             │   old.txt   │            new.txt            │
// ParsePrice    412.0n ± 2%    389.5n ± 1%   -5.46% (p=0.000 n=10)

The numbers a microbenchmark gives are real — for the microworld. A service adds queueing, contention, GC interference between requests, and cold caches. The bridge between worlds is the rule: microbenchmark to choose between implementations; load-test to make latency claims.

Load-testing honestly: percentiles and the open loop

Mean latency is the most useless number in a latency report: it is dominated by the fast majority and structurally blind to the tail your users actually feel. Report percentiles — p50, p99, p99.9 — and respect how tails compound: a page that fans out to 100 backend calls experiences a backend’s p99 on most page loads (the chance all 100 calls dodge the worst 1% is 0.99^100 ≈ 37%). Tail latency is not an edge case; at fan-out it is the median experience.

Then there is the Hook’s trap, coordinated omission. Closed-loop tools (a fixed pool of virtual users, each waiting for a response before the next request) measure service time but quietly stop arriving during stalls — precisely when arrivals would have queued and suffered. The recorded histogram omits the worst seconds in coordination with their occurrence. Open-loop tools (wrk2, vegeta, anything with a fixed arrival rate) keep firing on schedule; a request that arrives during a stall waits, and its full waiting time lands in the histogram. The distinction is not pedantry — it routinely changes p99 by 10–100×, which is the difference between shipping and paging. Rules of engagement: load the system at a fixed rate derived from real traffic (plus headroom), run long enough to cross several GC cycles and pool recycles (minutes, not seconds), and measure from the client side, including connection setup, the way users do.

The optimization loop: measure, hypothesize, change one thing, measure

Optimization without a loop degenerates into folklore. The senior workflow is mechanical: (1) measure under realistic load — open-loop, fixed rate, percentiles, with profiles captured during the run (a CPU or mutex profile from an idle service describes a different program); (2) form a falsifiable hypothesis — “p99 is GC assist during marking, driven by allocations in the serializer,” not “it’s probably the GC”; (3) change exactly one thing — the moment two changes ship together, the win is unattributable and the next regression undebuggable; (4) re-measure the same way, and let the numbers veto the narrative. Profiles under load are the hypothesis factory: the CPU profile says where cycles go at 5,000 rps, the mutex profile explains latency that CPU can’t, the heap alloc profile names the GC pressure. Two honesty rules from hard experience: beware optimizations that move cost rather than remove it (a sync.Pool that trades alloc churn for retention; a cache that fixes p50 and poisons p999 via eviction storms), and keep the load generator on separate hardware — a saturated generator measures itself.

Performance regressions in CI: benchstat or it didn’t happen

A single benchmark run is a coin flip wearing a lab coat. Run-to-run noise on shared CI runners is routinely ±5–10% — thermal state, neighbors, frequency scaling — so a pipeline that fails on “3% slower than last run” fails randomly, and a pipeline that compares single runs misses real 10% regressions hiding in noise. The honest setup: run each benchmark with -count=10 (or more), compare distributions with benchstat, which reports the delta with a p-value — and gate only on statistically significant changes beyond a threshold you chose deliberately (e.g., “fail on ≥5% with p < 0.05”). Pin what you can: dedicated or labeled runners, fixed CPU frequency where possible, GOMAXPROCS set explicitly, benchmarks isolated from parallel test jobs. For the numbers that actually page people — service p99 under load — a nightly open-loop load test against a staging deploy with archived profiles catches what per-function benchmarks structurally cannot: cross-cutting regressions in GC pressure, lock contention, and connection management. CI benchmarks guard functions; scheduled load tests guard the service.