Capstone: designing and hardening a production Node service
A production Node service is the whole track composed at once: typed errors behind one boundary, an unblocked loop, an ordered middleware stack with timeouts and graceful shutdown, validated input, env config, and observability. Each skipped concern is a named incident.
The prototype worked on the laptop, so it shipped. The first deploy under load wrote 200 OK to half the in-flight checkout requests and then dropped them — the pod got SIGTERM, the container ran the app as PID 1 with no signal handler, and Kubernetes hard-killed it after 30 seconds mid-transaction. The next incident was tail latency: one endpoint did JSON.parse on a 4 MB body inside the request path and blocked the event loop, so every other request queued behind it and p99 went from 40 ms to 9 seconds. Then a crafted query string reached an unparameterized SQL string. None of these were exotic. Each was one unit of this track that the prototype had skipped, and each became a production incident the moment real traffic arrived.
The checklist is the architecture
By the end of this lesson you will be able to trace a single request through a fully-hardened service and name the incident each skipped layer produces — so the next time a pull request cuts a corner, you’ll recognise it before it reaches production.
Going from prototype to production is not a rewrite — it is applying every unit of this track to one small HTTP service and refusing to skip any of them. The useful lens is a request-lifecycle checklist: for each cross-cutting concern, name what you do and the concrete signal for done when. Skip a row and you don’t get a smaller service; you get a future incident with a name. The 12factor manifesto and the Node best-practices repo are both, underneath, exactly this list — config in the environment, dependencies declared and locked, processes disposable, logs as event streams.
The senior move is to treat these as composing layers, not a feature backlog. Errors, the event loop, the middleware order, shutdown, validation, config, and observability are not independent checkboxes — they interlock. A graceful shutdown is worthless if the load balancer never learns you’re draining (that’s the readiness probe). An error boundary is worthless if a blocked loop means the boundary never runs. Below is the readiness checklist that this whole lesson defends.
| Concern (unit) | What you do | Done when |
|---|---|---|
| Errors (03) | Typed domain errors; one boundary maps type→status; process net on uncaught | Clients see safe JSON + codes; logs hold full stacks; a surprise crashes, not corrupts |
| Event loop (04) | Never block; offload CPU to a worker pool; watch loop lag; scale across cores | p99 stays flat under load; loop lag < ~50 ms; a profile is one command away |
| HTTP (05) | Ordered middleware; request + header + body timeouts; graceful SIGTERM | No request runs unbounded; a deploy drains in-flight work with zero dropped requests |
| Testing (06) | Unit tests for logic; integration tests (supertest + real test DB) for the wired API | CI gates merges on a green suite that exercises the real routes, not mocks |
| Security (07) | Validate input at the boundary; parameterized queries; secrets from env; npm ci + audit | No unvalidated input reaches logic; no secret in git; lockfile-pinned, audited deps |
| Packaging (08) | Multi-stage slim/distroless image, non-root, exec-form CMD + init, healthcheck | Small image runs as non-root and forwards signals; healthcheck reflects readiness |
| Config & ops (12-factor) | Config via env; liveness + readiness endpoints; metrics endpoint; SLOs | Same image runs every env via config; probes + metrics + traces exist; SLOs tracked |
The request lifecycle, hardened end to end
Trace one request and you can see every unit doing its job. The ingress or load balancer routes to a process whose HTTP server already has timeouts set — server.requestTimeout, headersTimeout, and a per-route handler timeout — so no single client can pin a connection open forever (unit 05). The request enters an ordered middleware stack, and order is load-bearing: logging first (so you log even rejected requests), then a body parser with a size cap (so a 4 MB body can’t blow up memory or block parse), then auth, then schema validation, then the handler. The handler calls a thin service layer, which calls a repository that issues parameterized queries — never string concatenation — and a domain failure becomes a typed error (unit 07, unit 03).
Threaded through all of it is request context. A pino structured logger carries a request/trace ID through AsyncLocalStorage, so every log line from deep in the stack is correlated to its request without passing the ID by hand. diagnostics_channel lets you publish instrumentation points (DB query start/end, cache hit) that exporters subscribe to without coupling. At the very end sits a single error boundary: error-handling middleware that maps error type to HTTP status (ValidationError→400, NotFoundError→404, anything unknown→500), sends a safe JSON body with no stack, and logs the full error object with its cause chain. That is unit 03’s strategy expressed as exactly one place in the wiring.
A CPU-bound step (image resize, bcrypt at high cost, big JSON transform) sits inside a hot request handler and is spiking p99 under load. Where do you put it?
Bootstrap and shutdown: the wiring that decides uptime
Two pieces of plumbing decide whether deploys are invisible or destructive: the bootstrap that installs the safety net before anything serves, and the shutdown that drains before exiting. When you look at a new service’s index.js or server.ts entry point, ask yourself: does it install the process net before listen, and does it handle SIGTERM before the first request can arrive? On SIGTERM you do not slam the server shut — you flip readiness to not ready (so the load balancer stops sending new requests), stop accepting connections, let in-flight requests finish, close the DB pool, and arm a hard timeout so a stuck request can’t block shutdown forever. Skipping this is the dropped-checkout incident from the hook.
import { createServer } from "node:http";
import { app } from "./app.js"; // ordered middleware + error boundary
import { logger } from "./logger.js";
import { closePool } from "./db.js";
let ready = false;
const server = createServer(app);
server.requestTimeout = 30_000; // unit 05: bound every request
server.headersTimeout = 10_000;
// unit 03: process net — record, then die; a supervisor restarts clean
process.on("uncaughtException", (err) => { logger.fatal({ err }); shutdown(1); });
process.on("unhandledRejection", (err) => { logger.fatal({ err }); shutdown(1); });
async function shutdown(code = 0) {
ready = false; // readiness flips → LB drains us
server.close(async () => { // stop accepting; finish in-flight
await closePool();
process.exit(code);
});
setTimeout(() => process.exit(code), 10_000).unref(); // hard cap
}
process.on("SIGTERM", () => shutdown(0));
server.listen(3000, () => { ready = true; logger.info("listening"); });
// liveness = process is up; readiness = `ready` flag (12-factor disposability)▸Why this works
Why does the readiness flip have to happen before server.close, not after? Because the load balancer learns you’re leaving only by polling the readiness endpoint. If you close the server first, new requests that the LB hasn’t yet stopped routing get a connection error instead of a clean drain. Flip readiness, give the probe interval a moment to notice, then close — that ordering is the difference between a zero-downtime deploy and a burst of 502s on every rollout.
A rolling deploy drops a handful of in-flight requests with connection-reset errors on every rollout. Which skipped concern is the cause?
You want every log line from deep in the call stack to carry the same request/trace ID without threading it through every function signature. What do you reach for?
Order the middleware/handler pipeline for one inbound request, from edge to data and back through the boundary:
- 1 Logging + request-ID assignment (so even rejected requests are recorded)
- 2 Body parser with a size cap (reject oversized payloads early)
- 3 Authentication / authorization
- 4 Schema validation of the parsed input
- 5 Route handler → service → repository (parameterized query)
- 6 Error boundary maps any thrown error type → status + safe body
- 01Walk the SIGTERM shutdown sequence and explain why the order matters for a zero-downtime deploy.
- 02Each row of the readiness checklist maps to a specific production incident if skipped. Name the incident for: no error boundary, blocked event loop, unvalidated input, and a root container that ignores SIGTERM.
A production Node service is this entire track composed into one small HTTP service, defended by a readiness checklist where every row maps to a concrete incident if skipped. Errors (unit 03): typed domain errors propagate with { cause }, a single error boundary maps type→status and logs the full object, and a process net on uncaughtException/unhandledRejection records then exits for a supervisor to restart clean. The event loop (unit 04) stays free — CPU work is offloaded to a worker pool, loop lag is monitored, and a CPU profile or heap snapshot is one command away when latency degrades. HTTP (unit 05) is a framework with an ordered middleware stack — logging → body-parser-with-size-cap → auth → validation → handler → error boundary — with request, header, and body timeouts and a graceful SIGTERM path that flips readiness, drains, closes, and hard-caps. Testing (unit 06) gates CI on unit tests plus integration tests against a real test DB. Security (unit 07) validates all input at the boundary, uses parameterized queries, loads secrets from the environment, and runs npm ci + audit with a pinned lockfile. Packaging (unit 08) ships a multi-stage slim/distroless image running non-root with an exec-form CMD + init so signals reach the process. And config/ops follows 12factor: config via env, liveness and readiness probes, a metrics endpoint, logs/metrics/traces, and SLOs. The whole point: production-readiness is not extra features bolted on later — it is these layers interlocking. Now when you review a service and spot a missing SIGTERM handler, an inline bcrypt call, or a body parser with no size cap, you know exactly which incident that corner-cut has already booked.
Practice
Start at the top. Tasks go easiest → hardest: recall a fact, apply it to a case, then a senior-level stretch. Open one, attempt it, then reveal.
Something unclear?
Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.