Goroutines and the scheduler: cheap stacks, GMP, and the leaks that outlive requests

Why a goroutine costs ~2 KB and a thread costs megabytes

An OS thread is expensive in two ways: its stack is reserved up front (8 MB virtual by default on Linux — check ulimit -s), and every block/wake is a kernel context switch costing roughly 1–2 µs plus cache pollution. A goroutine starts with a ~2 KB stack that the runtime grows and shrinks on demand: each function prologue checks for headroom, and on overflow the runtime allocates a bigger segment and copies the stack over (contiguous stacks, since Go 1.4 — pointers into the stack are rewritten during the copy). Switching between goroutines is a user-space register swap on the order of tens of nanoseconds, no kernel involved. That three-orders-of-magnitude gap is the whole pitch: 10,000 threads is an incident, 10,000 goroutines is a Tuesday. The honest cost: each goroutine is invisible to the OS, so OS tools (top, thread counts) tell you nothing about how many you have — only the runtime knows.

GMP: how the scheduler actually places work

The runtime scheduler juggles three entities. G is a goroutine — the stack plus the saved registers. M is an OS thread the runtime owns. P is a processor — a scheduling token holding a local run queue of Gs; there are exactly GOMAXPROCS of them (default: number of CPU cores, and since Go 1.25 the default also respects cgroup CPU limits in containers — before that, a 2-CPU-quota pod on a 64-core node got GOMAXPROCS=64 and throttled itself, the classic fix being uber’s automaxprocs). An M must hold a P to run Go code. The choreography:

• A new G goes into the current P’s local run queue (256 slots); overflow spills to a global queue.
• An idle P with an empty queue steals half the run queue of a random other P — work stealing keeps cores busy without a central lock.
• When a G blocks in a syscall, its M is stuck in the kernel — so the runtime hands the P to another M (parking or spawning one) and the remaining Gs keep running. This is why one slow disk read does not stall the other 9,999 goroutines.
• Blocking on a channel or mutex is cheaper still: the G is parked in the runtime (no thread blocks at all), and the M picks the next G from the queue.

// Observing the machinery: a goroutine per task, bounded by GOMAXPROCS cores.
func main() {
	fmt.Println("GOMAXPROCS =", runtime.GOMAXPROCS(0)) // 0 = read, don't set

	var wg sync.WaitGroup
	for i := 0; i < 100_000; i++ { // 100k goroutines: ~200 MB of stacks, fine
		wg.Add(1)
		go func(id int) {
			defer wg.Done()
			work(id) // runtime schedules these across GOMAXPROCS Ps
		}(i)
	}
	wg.Wait()
	fmt.Println("still alive:", runtime.NumGoroutine()) // should be ~1
}

Preemption: cooperative until 1.14, then signals

Before Go 1.14 the scheduler was cooperative: a goroutine yielded only at function calls (where the stack check lives). A tight loop — for { x++ } — never yielded, so it could pin a P forever, starve the garbage collector’s stop-the-world phase, and freeze the program. Since Go 1.14, the runtime’s monitor thread (sysmon) notices a G running longer than ~10 ms and sends the M a signal (SIGURG on Unix); the handler interrupts the goroutine at the next safe instruction. Loops without calls can no longer wedge the runtime. The remaining sharp edge: preemption is still not instant or free — a CPU-bound goroutine gets ~10 ms slices, so 8 hot loops on GOMAXPROCS=8 will still make every latency-sensitive goroutine wait its turn. CPU-heavy work belongs in a bounded pool, not in unbounded go statements.

The failure mode: goroutine leaks, and pprof as the census

A goroutine leak is a goroutine that will never resume: blocked on a channel no one will send to, a receive no one will close, a ctx.Done() it never selects on, or a mutex held by a goroutine that exited. The garbage collector cannot collect it — a parked goroutine is a GC root, so its entire stack and everything it references stays live. Leaks accrete: one per failed request times a million requests is the Hook’s slope. Detection is the goroutine profile:

import _ "net/http/pprof" // exposes /debug/pprof/ on the default mux

// then: go tool pprof http://localhost:6060/debug/pprof/goroutine
// or:   curl localhost:6060/debug/pprof/goroutine?debug=1
// The output groups goroutines by identical stack — a leak shows up as
// one stack with an absurd count, e.g.:
//   1184202 @ 0x43a1c5 ... chan receive
//     github.com/acme/gw/internal/fanout.collect+0x9e

Read it like a histogram: thousands of goroutines parked on the same line means whoever spawns them never arranged their exit. The cure is structural, not cosmetic — every go statement should answer “how does this goroutine end?”: a context it selects on, a channel that gets closed, a WaitGroup someone waits for. Watch runtime.NumGoroutine() on a dashboard; a counter that only goes up is a leak in progress. Honest numbers: a parked goroutine costs ~2–4 KB of stack plus whatever its closure captured (often the real cost — a captured 64 KB buffer turns a million leaks into 64 GB).