Fuzzing and benchmarks: coverage-guided inputs, b.Loop, and benchstat honesty

Native fuzzing: seeds, mutation, and the corpus

Since Go 1.18 fuzzing is built into go test. A fuzz target is a function FuzzXxx(f *testing.F): you register seed inputs with f.Add, then hand f.Fuzz a property that must hold for any input:

func FuzzParseInterval(f *testing.F) {
	f.Add("90s")
	f.Add("2h45m")
	f.Add("")
	f.Fuzz(func(t *testing.T, s string) {
		d, err := ParseInterval(s)
		if err != nil {
			return // invalid input may be rejected — it must not panic
		}
		// round-trip property: format what we parsed, parse it again
		s2 := FormatInterval(d)
		d2, err := ParseInterval(s2)
		if err != nil || d2 != d {
			t.Fatalf("round-trip broke: %q -> %v -> %q -> %v (%v)", s, d, s2, d2, err)
		}
	})
}

Without -fuzz, go test just runs the seeds plus the committed corpus — cheap, deterministic, CI-friendly. With go test -fuzz=FuzzParseInterval, the engine mutates inputs coverage-guided: an input that reaches a new branch is kept and mutated further, so the search concentrates on code it has not seen, typically tens of thousands of execs per second per worker. The mechanism is why fuzzing finds what tables cannot: a table encodes examples its author imagined; the fuzzer explores the input space your code actually branches on — sign-without-digits, lone UTF-8 continuation bytes, 1e309. When an input fails, the engine minimizes it (shrinking while the failure reproduces) and writes it to testdata/fuzz/FuzzParseInterval/ — commit that file and the crash is a permanent regression test, run by ordinary go test with no flags. The craft is in the property: panics come free, but “no panic” is weak; round-trips (parse∘format = id), comparisons against a reference implementation, and invariants (output length, sortedness) are where fuzzing earns senior-grade value.

Benchmarks: b.Loop, ns/op, allocs/op

When you read a benchmark result, how do you know you are looking at the cost of your function and not the cost of an optimized-away loop? That question has a precise answer — and it changed in Go 1.24.

A benchmark is BenchmarkXxx(b *testing.B), run via go test -bench=.. Since Go 1.24 the loop is b.Loop():

func BenchmarkParseInterval(b *testing.B) {
	b.ReportAllocs()
	input := strings.Repeat("1h30m", 1) // setup, outside the loop
	for b.Loop() {
		d, err := ParseInterval(input)
		if err != nil {
			b.Fatal(err)
		}
		_ = d
	}
}

b.Loop fixes two chronic lies of the older for i := 0; i < b.N; i++ style. First, dead-code elimination: the compiler may notice a result is never used and delete the call — old benchmarks needed package-level sink variables to defeat it; b.Loop keeps the loop body’s arguments and results alive automatically. A benchmark reporting 0.25 ns/op is reporting the cost of an empty loop, not your function. Second, setup amortization: the b.N style re-ran the whole function with growing N (1, 100, 10000…) until the run exceeded -benchtime (default 1 s), re-executing setup each round unless you remembered b.ResetTimer; b.Loop runs setup exactly once and times only the loop. The numbers: ns/op is wall time per iteration; -benchmem or b.ReportAllocs() adds B/op and allocs/op — allocation counts are exact (from the runtime’s allocator counters) and usually the most stable, most actionable number a benchmark produces: cutting allocs/op from 7 to 1 is a real win regardless of timer noise.

benchstat: statistical honesty

A single before/after run proving your optimization made things “4% faster” proves nothing. Wall-clock benchmarks on real machines wobble — thermal throttling, core migration, cache and ASLR layout, a browser tab — easily ±2–5% on a laptop, worse on shared CI runners. The honest protocol: run each side ~10 times and let benchstat do the statistics:

// go test -bench=ParseInterval -count=10 > old.txt
// ... apply the change ...
// go test -bench=ParseInterval -count=10 > new.txt
// benchstat old.txt new.txt
//
//             │   old.txt   │             new.txt              │
// ParseInterval  412.5n ± 2%   369.0n ± 1%   -10.55% (p=0.000 n=10)
// (an insignificant change prints: ~ , p=0.483)

benchstat reports the median, the spread, and a p-value from a Mann-Whitney U test; if the distributions are statistically indistinguishable it prints ~ instead of a delta. That tilde is the feature: it stops you from shipping noise as a win. Discipline that keeps the numbers real: quiet machine or dedicated runner, -count=10, compare allocs/op first (exact), and benchmark the level users feel — a 30% win in a function that is 0.1% of the profile is a rounding error, which is why benchmarks pair with pprof, not replace it.