Healthchecks and lifecycle: liveness vs readiness vs startup, and the shutdown sequence that drops zero requests

Three probes, three different questions

By the end of this lesson you will know exactly which of the three probes belongs to which question — and why getting that wrong turns a 28-second blip into a 40-minute outage every time.

Liveness asks: is this process broken beyond self-repair — deadlocked, wedged, corrupted? Failure means the kubelet restarts the container, so the handler must test only conditions a restart fixes: the event loop responds, period. It must never check dependencies, because a restart does not repair a database — it just converts every dependency outage into a fleet-wide restart storm, exactly the Hook. Readiness asks: should this pod receive traffic right now? Failure removes the pod from Service endpoints — reversible, cheap, exactly what you want during a dependency outage, an overload, or a drain. This is where dependency checks belong. Startup asks: has the process finished booting? Until it succeeds, liveness and readiness are suspended — the grace budget for migrations and cache warms is failureThreshold times periodSeconds, and without it liveness kills a slow boot at its threshold and loops forever.

livenessProbe:
  httpGet: { path: /healthz, port: 8080 }
  periodSeconds: 10
  failureThreshold: 3        # 30 s of true wedge before restart
readinessProbe:
  httpGet: { path: /readyz, port: 8080 }
  periodSeconds: 5
  failureThreshold: 2        # out of rotation within ~10 s
startupProbe:
  httpGet: { path: /healthz, port: 8080 }
  periodSeconds: 5
  failureThreshold: 36       # up to 180 s to boot before liveness arms

Wiring real readiness in Go

A readiness handler that returns 200 unconditionally is theater — it tells Kubernetes a pod is serviceable when the pod has no idea. Real readiness checks what the service actually needs to serve every request, with a timeout, and carries a drain flag:

var ready atomic.Bool // set true once boot completes

mux.HandleFunc("GET /healthz", func(w http.ResponseWriter, r *http.Request) {
	w.WriteHeader(http.StatusOK) // liveness: process alive — nothing else
})

mux.HandleFunc("GET /readyz", func(w http.ResponseWriter, r *http.Request) {
	if !ready.Load() {
		http.Error(w, "draining", http.StatusServiceUnavailable)
		return
	}
	ctx, cancel := context.WithTimeout(r.Context(), 1*time.Second)
	defer cancel()
	if err := db.PingContext(ctx); err != nil { // the dependency every request needs
		http.Error(w, "db unreachable", http.StatusServiceUnavailable)
		return
	}
	w.WriteHeader(http.StatusOK)
})

Discipline cuts both ways: only hard dependencies belong here — the ones whose absence fails every request. Wiring an optional cache or a non-critical downstream into readiness means one flaky sidecar removes your whole fleet from rotation; optional dependencies degrade responses, they do not unready pods. And keep the endpoint taxonomy straight: /healthz is trivial and for the kubelet, /readyz checks dependencies and gates traffic, /debug/pprof is for engineers and never public — importing net/http/pprof registers handlers on DefaultServeMux as a side effect, which is how profilers end up internet-facing; serve it on a separate localhost or cluster-internal listener.

The shutdown sequence and the grace budget

Termination is a distributed handshake, and the probes are half of it. On SIGTERM, endpoint removal propagates asynchronously — load balancers keep sending new requests for several seconds. A server that stops accepting immediately serves connection-refused to live traffic on every deploy. The correct order: fail readiness first, wait out propagation, then drain — this spirals straight into the go/03 graceful-shutdown lesson, now with the cluster choreography attached:

<-ctx.Done()                 // SIGTERM from the kubelet
ready.Store(false)           // 1. /readyz now 503 — endpoint removal begins
time.Sleep(5 * time.Second)  // 2. absorb endpoint propagation lag
shutdownCtx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
defer cancel()
_ = srv.Shutdown(shutdownCtx) // 3. drain in-flight requests

The arithmetic is non-negotiable: terminationGracePeriodSeconds (default 30) must cover the propagation sleep plus the drain deadline plus headroom, because when it expires SIGKILL arrives unconditionally and takes the unfinished drain with it. Five plus twenty leaves five seconds of headroom inside the default — and if your slowest legitimate request runs 25 seconds, the default budget is already broken: raise the grace period to 40, do not shave the sleep. Probe timing folds into the same arithmetic from the other side: readiness flips within failureThreshold times periodSeconds, so 2 x 5 s means the LB starts removing you about ten seconds after the flip in the worst case — the sleep covers the controller-to-dataplane lag, the probe cadence covers detection.