Zero-downtime deploys: health probes, draining, and rollouts

The deploy-time 502 is a race, not a crash

You already know (from containerizing Node) that on a rollout the orchestrator sends SIGTERM, you handle it, call server.close(), and exit before terminationGracePeriodSeconds runs out. That makes your pod shut down cleanly. It does not make the deploy clean, because shutting down cleanly and being removed from the load balancer are two independent events that race.

Here is the exact sequence on a naive rolling deploy. The orchestrator decides to replace pod A. It does two things, and crucially they are not synchronized: it sends SIGTERM to pod A’s container, and—separately, through the endpoints/service controller and then the load balancer or kube-proxy on every node—it begins removing pod A from the set of endpoints traffic is routed to. The first is near-instant. The second propagates through a control plane: the endpoint update has to be observed, then the LB or every node’s iptables/IPVS rules have to be rewritten. That propagation takes hundreds of milliseconds to several seconds.

So for that window, pod A is in the worst possible state: it received SIGTERM, your handler called server.close() (which stops accepting new connections), but the LB still has pod A in rotation and keeps opening new connections to it. New connections to a socket that’s no longer accept()ing get refused — the client sees ECONNREFUSED/ECONNRESET, or the LB returns a synthetic 502/503. Multiply by every deploy: a burst of 502s for ~2–10 seconds per rollout, every single rollout, scaling with your request rate. The failure mode is invisible in any single request log and obvious only in the aggregate error-rate graph — which is exactly why it survives for months.

The fix is to reverse the order so the LB stops routing before you stop accepting. You don’t get to synchronize the two events, but you can insert a delay that makes de-registration win the race.

Liveness vs readiness: two probes, two completely different jobs

Kubernetes gives you two health probes that beginners conflate and seniors keep rigidly separate, because they answer different questions and have opposite consequences.

• Liveness probe — “is this process wedged? restart it if so.” A failing liveness probe kills the pod (it gets SIGTERM, then SIGKILL, then a fresh container starts). It is a self-heal for a hung event loop or a deadlock.
• Readiness probe — “should traffic go here right now?” A failing readiness probe removes the pod from the Service endpoints so the LB stops routing to it — but it does not kill the pod. It’s a routing switch, not a kill switch.

That distinction is the lever that fixes the deploy 502. On SIGTERM, fail readiness first: flip an in-memory flag so the readiness endpoint returns 503. The endpoint controller observes the failure and pulls you from rotation. Then wait long enough for that de-registration to actually propagate, and only then call server.close() and drain in-flight work. You’ve manually ordered the race in your favor.

import http from "node:http";

let ready = true;             // readiness flag, flipped on shutdown
const inflight = new Set();   // optional: track sockets for diagnostics

const server = http.createServer((req, res) => {
  if (req.url === "/healthz") {          // LIVENESS: process responds → alive
    res.writeHead(200).end("ok");        //   no DB call here — see the inset
    return;
  }
  if (req.url === "/ready") {             // READINESS: route to me?
    res.writeHead(ready ? 200 : 503).end(ready ? "ready" : "draining");
    return;
  }
  // ... real request handling ...
  res.writeHead(200).end("hello");
});

server.listen(3000);

process.on("SIGTERM", async () => {
  ready = false;                          // 1. fail readiness → LB drains us
  await sleep(LB_DEREGISTER_MS);          // 2. wait for de-registration to propagate
  server.close(() => process.exit(0));    // 3. stop accepting, finish in-flight, exit
  server.closeIdleConnections?.();        //    evict idle keep-alive sockets now
  setTimeout(() => process.exit(1), DRAIN_DEADLINE_MS).unref(); // 4. hard cap
});

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

The tradeoff lives in what each probe is allowed to depend on. Readiness should reflect real dependencies — if the DB pool is exhausted, returning 503 and shedding traffic is correct backpressure. Liveness must not — and that asymmetry is the single most expensive probe mistake in production.

Draining in-flight requests, and the keep-alive trap

server.close() does exactly two things: it stops accepting new connections, and it fires its callback once all existing connections have finished and closed. That sounds like clean draining — and for short request/response traffic it is. The trap is HTTP keep-alive: a persistent connection that’s idle (no request in flight) is still an open connection, so server.close() will wait for it indefinitely. A client holding a keep-alive socket open for connection reuse can pin your shutdown until the grace period expires.

That’s why a correct drain has a deadline and actively evicts idle sockets. Node 18.2+ added server.closeIdleConnections() (evict sockets with no in-flight request) and server.closeAllConnections() (the hammer — kill everything). The shape: on SIGTERM, after failing readiness, call server.close(cb) to begin draining, immediately closeIdleConnections() to release idle keep-alive sockets, and arm a hard-deadline timer that force-exits if in-flight work runs long. Set server.headersTimeout/keepAliveTimeout so sockets don’t outlive requests in the first place.

The numbers have to nest correctly or you get cut off mid-request:

# Deployment — the timing budget that prevents both 502s and SIGKILL-mid-request
spec:
  template:
    spec:
      terminationGracePeriodSeconds: 40   # MUST exceed preStop + drain, or you get SIGKILL (137)
      containers:
        - name: api
          lifecycle:
            preStop:
              exec:
                command: ["sleep", "5"]    # bridge the LB-deregistration race
          readinessProbe:
            httpGet: { path: /ready, port: 3000 }
            periodSeconds: 2                # detect "draining" within ~2s
            failureThreshold: 1
          livenessProbe:
            httpGet: { path: /healthz, port: 3000 }  # NO dependency check
            periodSeconds: 10
            failureThreshold: 3             # ~30s before a restart — tolerant, not trigger-happy

terminationGracePeriodSeconds (k8s default 30s) is the total budget from SIGTERM to SIGKILL. Your preStop sleep (5s) plus your in-app drain (15–25s) must fit inside it with margin — if drain overruns the grace period, the kubelet sends SIGKILL (exit 137) straight through your still-running requests, dropping them. That’s the failure mode of setting the grace period too low: you traded one source of 502s (the LB race) for another (the kill saw). The preStop sleep 5 is a belt-and-suspenders bridge for the de-registration window even if your app’s readiness flip is slow to propagate.

Rollout strategies: who eats a bad deploy

Probes and draining make a single pod swap clean. The rollout strategy decides how many pods swap at once, how fast you can undo it, and—critically—who is exposed when the new version is broken in a way no probe catches (a logic bug, a bad config, an incompatible migration).

The failure mode that ties everything together is the non-backward-compatible migration. During any rolling or canary deploy, the old and new versions run simultaneously against one shared database. If the new version’s migration renames or drops a column the old version still reads, the old (still-serving) pods start throwing — your “zero-downtime” deploy just broke the version you didn’t even change. The discipline is expand/contract: first deploy a migration that only adds (a new nullable column, a new table) and code that writes both old and new shapes; let it bake; then, in a later deploy, contract by removing the old column once no running version reads it. Every intermediate schema must be readable by both the version you’re rolling out and the one you’re rolling away from. Skip expand/contract and your perfectly drained, probe-gated, canaried rollout still takes an outage — just from the database side.