Graceful shutdown: Shutdown(ctx), SIGTERM wiring, and why deploys still drop requests

What Shutdown(ctx) actually does

srv.Shutdown(ctx) executes a precise sequence: close all listeners (new connects get refused), close idle keep-alive connections, then wait — polling — for active requests to finish. It returns when the server is fully drained, or when the ctx expires, whichever comes first. On expiry it returns ctx.Err() but does not forcibly kill in-flight connections; pair it with srv.Close() if you need a hard stop after the grace period. Meanwhile the blocked ListenAndServe call returns immediately with http.ErrServerClosed — expected, not an error:

srv := &http.Server{Addr: ":8080", Handler: mux}

go func() {
	if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
		log.Fatalf("listen: %v", err)
	}
}()

<-shutdownSignal // see next section

shutdownCtx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
if err := srv.Shutdown(shutdownCtx); err != nil {
	log.Printf("drain incomplete: %v", err) // ctx expired with requests in flight
	srv.Close()                             // hard-close the stragglers
}

Two details seniors get asked about. First, ordering: Shutdown must run on a different goroutine than ListenAndServe, because the latter is blocked serving. Second, the drain deadline is a policy number: it must be at least your slowest legitimate request, and at most what your platform allows — Kubernetes sends SIGKILL after terminationGracePeriodSeconds (default 30 s), and SIGKILL is not a conversation.

Wiring the signal: signal.NotifyContext

The process learns about its death from a signal — SIGTERM from Kubernetes/systemd, SIGINT from a human’s Ctrl-C. Since Go 1.16 the idiomatic bridge from signal to cancellation is one line:

ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM, os.Interrupt)
defer stop()

// ... start server ...

<-ctx.Done() // blocks until SIGTERM/SIGINT
stop()       // restore default handling: a SECOND signal kills immediately

NotifyContext returns a context canceled on the first matching signal. The stop() after Done() is a deliberate operability move: it unregisters the handler, so a second Ctrl-C bypasses your drain and kills the process — the escape hatch for a shutdown that itself hangs. Without that, an impatient operator mashing Ctrl-C is held hostage by your 30-second drain. One more sharp edge: while you drain, in-flight handlers still run — if they check a global “shutting down” flag or you cancel their base context too early, you turn a graceful drain into self-inflicted 500s. Drain means let them finish, with their own deadlines as the bound.

What Shutdown does not cover

Before you declare your shutdown logic complete, ask: does this service have websockets or long-lived streams? If yes, Shutdown alone is not enough.

Shutdown waits for connections in the normal HTTP request cycle. Two classes of traffic are explicitly outside it:

• Hijacked connections — anything that took the raw TCP conn via http.Hijacker, which is every websocket. After hijack, net/http no longer tracks the connection’s state; Shutdown neither waits for it nor closes it.
• Long-lived streams that never end on their own — an SSE handler in a for loop is an active request, so a plain Shutdown will wait for it until the drain deadline, every deploy, unless you tell it to wind down.

The hook for both is srv.RegisterOnShutdown(f): each registered function runs (in its own goroutine) when Shutdown starts. Use it to broadcast “closing” to websocket sessions, cancel the streams’ context, and give them a beat to send a close frame:

srv.RegisterOnShutdown(func() {
	hub.CloseAll(1 * time.Second) // websocket close frames, then drop
	streamCancel()                // SSE loops select on this ctx and exit
})

Without this, a deploy either snaps every websocket mid-frame (clients see abnormal closure, reconnect storms follow) or hangs the drain until SIGKILL does the snapping for you — at which point in-flight HTTP requests die too, the exact thing Shutdown was for.

Kubernetes: stop the traffic before you stop the server

In a cluster, Shutdown is the second step of the dance, and running it first causes the Hook’s 502s. When a pod enters Terminating, two things happen in parallel: the kubelet sends SIGTERM (after preStop), and the endpoints controller starts removing the pod from Endpoints — but ingresses and kube-proxy rules converge with a lag of seconds. Close your listener during that lag and you bounce live traffic. The robust sequence:

• Fail readiness immediately on SIGTERM (flip a flag your /readyz handler reads). Readiness controls membership in load balancing; liveness does not — failing liveness just gets you restarted.
• Wait out propagation — a preStop hook with sleep 5 (or an in-process sleep before Shutdown) covers typical endpoint convergence; clusters with slow ingress controllers need more.
• Then Shutdown(ctx) with a deadline that fits inside terminationGracePeriodSeconds — leave headroom: grace 30 s, preStop 5 s, drain ≤ 20 s, buffer 5 s.

The numbers worth memorizing: SIGKILL is unconditional at the end of the grace period; new-connection refusal during propagation shows up as 502/503 at the ingress at roughly (propagation lag × request rate) failures per deploy — at 1,000 rps and 3 s of lag, that is ~3,000 failed requests per pod, which is exactly the difference between “nobody noticed the deploy” and a postmortem.