Deadlock postmortems: lock-order inversion, channel-under-mutex, and reading the goroutine dump

Shape A: lock-order inversion

The Hook’s shape, and the classic. Two locks, two code paths, opposite orders:

func Transfer(from, to *Account) { // path 1: from → to
	from.mu.Lock()
	defer from.mu.Unlock()
	to.mu.Lock() // goroutine 412 waits here, holding from.mu
	defer to.mu.Unlock()
	// ...
}

func Audit(a, b *Account) { // path 2 locks the same pair in reverse
	b.mu.Lock()
	a.mu.Lock() // goroutine 9817 waits here, holding b.mu
	// ...
}

Each goroutine holds what the other needs: a cycle of length two, frozen forever — sync.Mutex has no timeout and no owner tracking. The window is tiny (both must be mid-acquisition in the same microseconds), which is why this ships: it survives code review, tests, and months of production before the interleaving lands. The fix is not cleverness, it is convention: define one global lock order and acquire in that order on every path. For dynamic pairs like accounts, order by an intrinsic key — if from.ID > to.ID { from, to = to, from } before locking — and write the convention down where the next engineer will trip over it.

Shape B: a channel operation under a mutex

func (s *Store) Flush() {
	s.mu.Lock()
	defer s.mu.Unlock()
	s.events <- snapshot(s.data) // send while holding s.mu — the receiver...
}

func (s *Store) consume() {
	for ev := range s.events {
		s.mu.Lock() // ...needs s.mu before it can finish receiving the NEXT one
		apply(ev)
		s.mu.Unlock()
	}
}

Flush sends while holding the lock; the consumer needs that same lock to process. One buffered slot of slack and it works in every demo — until the buffer is full at the same moment the consumer is between receive and lock. Then Flush waits for the consumer, the consumer waits for the mutex, cycle closed. The rule generalizes beyond channels: never perform a blocking operation — channel send or receive, I/O, RPC, Wait — while holding a mutex. A mutex is for nanosecond-scale memory edits; anything that can park the goroutine turns the critical section into a wait-graph node other goroutines may already depend on. Copy the data under the lock, release, then send.

Shape C: WaitGroup misuse

The third postmortem is subtler because it is a race that sometimes wears a deadlock costume:

var wg sync.WaitGroup
for _, job := range jobs {
	go func() {
		wg.Add(1) // WRONG: races with wg.Wait below
		defer wg.Done()
		process(job)
	}()
}
wg.Wait() // may observe counter == 0 before any Add ran

Add must happen before Wait can observe the counter — which means before the go statement, in the spawning goroutine. Written as above, the scheduler decides the outcome: if Wait runs before the children get scheduled, it sees zero and returns — work silently lost, or worse, main exits and the children are reaped mid-write. The mirror image deadlocks instead: a child calls wg.Wait() (or sends to an unbuffered channel read after Wait) while the parent holds something the child needs. Related classics in the same family: passing a WaitGroup by value (the copy’s Done never reaches the original — Wait blocks forever) and reusing a WaitGroup for a second wave before the first Wait returned. go vet catches the copy; -race reliably flags Add-racing-Wait.

The detection toolbox

The runtime’s famous fatal error: all goroutines are asleep - deadlock! is nearly useless in services: it fires only when every goroutine is blocked on runtime-visible operations — one live accept loop or ticker suppresses it. Production deadlocks are partial, and the instrument is the goroutine profile:

import _ "net/http/pprof"

// live dump with stacks and wait durations:
//   curl localhost:6060/debug/pprof/goroutine?debug=2
//
// goroutine 412 [sync.Mutex.Lock, 16 minutes]:   ← held-for duration = smoking gun
//   main.Transfer ... transfer.go:42
// goroutine 9817 [sync.Mutex.Lock, 16 minutes]:
//   main.Audit ... audit.go:31
// 9412 @ ... sync.Mutex.Lock main.(*Handler).Charge   ← the victim cluster

Read it like a detective, in three passes. First, cluster sizes (debug=1 groups identical stacks): 9,412 goroutines on one Lock line is the victim pile and names the contested lock. Second, wait durations (debug=2): minutes-long semacquire or chan send waits separate a deadlock from mere contention, which churns. Third, find the holders: the one or two goroutines blocked on a different lock than the cluster — they hold what the cluster wants and want what each other holds; their two stack traces are your cycle, file and line. Complete the kit with go test -race for the racy cousins (Shape C), go vet for WaitGroup copies, and the mutex profile (runtime.SetMutexProfileFraction + /debug/pprof/mutex) to watch contention trend toward the cliff before you fall off it. Together, these three passes turn a mysterious timeout into a file and line number in under ten minutes — without restarting the process or losing the evidence.

Prevention: boring rules that survive 3 a.m.

Every rule below is one of the three postmortems, inverted. Keep critical sections tiny — lock, touch memory, unlock; compute and I/O happen outside. Never block under a mutex: no channel ops, no RPC, no Wait — copy under the lock, act after release. One documented lock order per package (or an intrinsic-key ordering for dynamic pairs), enforced in review because no tool enforces it for you. Add before go, never inside the child; pass WaitGroups by pointer; run -race in CI always — it catches Shape C and a hundred relatives. And keep net/http/pprof wired in production before the incident: a deadlock you can dump in one curl is a 5-minute diagnosis, and the same dump taken after the restart shows nothing at all.