Health and lifecycle: honest probes, the SIGTERM grace window, and preStop

Three probes, three different questions

The cardinal mistake is one health check answering “is the process up?” when production needs three different answers. Docker’s HEALTHCHECK and Kubernetes’ probes formalize the distinction:

HEALTHCHECK --interval=10s --timeout=2s --start-period=30s --retries=3 \
  CMD curl -f http://localhost:8080/healthz || exit 1
#   --interval: run every 10s   --retries=3: 3 consecutive fails → unhealthy
#   --start-period=30s: failures during the first 30s don't count (slow boot grace)

• Liveness answers “is this process wedged and unrecoverable?” — if it fails, the orchestrator restarts the container. Make it cheap and dependency-free: a wedged event loop or a deadlocked process should fail it, but a transient database blip should not, or you turn a downstream outage into a restart storm that makes everything worse.
• Readiness answers “should this instance receive traffic right now?” — if it fails, the load balancer stops routing to it but does not restart it. This is where you check real dependencies: a 200 ms query against the connection pool, as in the Hook. A pod that loses its database should go NotReady (drain traffic) without going non-live (restart won’t fix a database outage).
• Startup answers “has the slow boot finished?” — it gates the other two during initialization so a JVM that takes 40 s to warm up isn’t liveness-killed at second 10. The Docker equivalent is --start-period.

The dangerous default is a probe that passes while the app is wedged — a TCP-port check, or a /healthz that returns 200 from a static handler that never touches the work path. A health check is only honest if it can go red for the same reason a user’s request would fail. The classic failure mode runs the other way too: a liveness probe that calls the database turns a 30-second database hiccup into every pod being restarted at once, deleting your warm caches and connection pools exactly when the system is already stressed.

The shutdown lifecycle: SIGTERM, grace, SIGKILL, preStop

Getting the probe split right buys you nothing if the shutdown sequence drops requests on the way out. That is the second half of honest lifecycle design.

A rollout terminates old pods, and how it does that decides whether in-flight requests survive. The contract is a fixed sequence. When the orchestrator decides to stop a pod it (1) optionally runs a preStop hook, then (2) sends SIGTERM to PID 1, then (3) waits up to terminationGracePeriodSeconds (default 30 s), and finally (4) sends SIGKILL to anything still running. Your job is to fully drain within the grace window so step 4 never fires on a live request.

lifecycle:
  preStop:
    exec: { command: ["sh", "-c", "sleep 5"] }   # bridge the LB de-registration race
terminationGracePeriodSeconds: 45                # > worst-case drain (p99 + LB propagation)

The subtle bug is a race that has nothing to do with your signal handler. The moment a pod is marked Terminating, two things happen concurrently: SIGTERM is delivered, and the endpoint is removed from the Service. But endpoint removal propagates asynchronously to every kube-proxy and load balancer — for a second or two, traffic is still being routed to a pod that has already started shutting down. If your SIGTERM handler closes the listener immediately, those in-transit requests hit a closed socket and get connection-refused. The fix is the preStop hook: a short sleep (commonly 5–15 s) that delays SIGTERM, keeping the pod fully serving while the de-registration propagates, so no new traffic is dropped. Then SIGTERM arrives, your handler flips readiness off (belt and braces), drains in-flight requests, and exits 0 — all comfortably inside a grace period sized above your p99 plus that propagation delay. Set the grace too low and SIGKILL severs live connections mid-response (the client sees an RST); forget preStop and the LB race drops requests even with a perfect handler.