Request lifecycle: instrument and harden a service

Build a small HTTP service (Go, Node, or JVM) behind a reverse proxy, instrument all seven lifecycle stops, then deliberately trigger one failure at each stop and harden it — proving every fix with measurements under identical load.

• Build a service with at least: one allocation-heavy JSON endpoint that returns an unbounded query result, one streaming/export endpoint that writes a large body, and one endpoint that fans out to 2-3 (possibly mocked) downstream calls. Put nginx or Envoy in front.
• Instrument the lifecycle: request-ID + per-stage timing (accept-wait, parse, route, middleware, handle, serialize, drain), accept-queue depth and overflow counters, response-size and serialization time, writable-stream backpressure events, and outbound-call latency with timeouts. Surface these as logs or metrics panels.
• Accept: lower somaxconn / the listen backlog (or slow the handler) and load-test until connections are refused with empty application logs. Capture the overflow, then fix by raising the backlog and shedding load gracefully; show connections are no longer silently dropped.
• Parse + Route + Middleware: reproduce a 431 from oversized headers and a middleware-ordering bug (a protected route reachable without auth, or a body parser that buffers before a size limit). Fix the header cap and reorder the chain so protective layers run first; prove the route is now guarded and oversized bodies are rejected before buffering.
• Serialize: load the unbounded-JSON endpoint until JSON.stringify dominates CPU and stalls other requests. Fix with pagination/LIMIT and a schema serializer; show serialization time and p99 drop.
• Drain: stream the large body to an artificially slow client (throttled socket) with backpressure ignored, and watch RSS climb toward OOM. Fix with pipeline()/'drain' handling and show memory stays flat at ~the high-water mark.
• Timeouts/tail: give every outbound call an explicit timeout, propagate a request deadline (min(local, remaining)) down the fan-out, and run the calls concurrently. Load-test the fan-out and record the p99 before and after; optionally add hedging at ~p95.

• A before/after table per stop: the failure symptom, the metric that exposed it (overflow count, 431 rate, serialization ms, RSS, p99), and the value after the fix — all measured under identical load, not estimated.
• The accept-queue overflow no longer produces silent drops, the protected route is unreachable without auth, oversized headers/bodies are rejected before buffering, and the streaming endpoint holds flat memory under a slow client.
• The fan-out endpoint's p99 is bounded by the propagated deadline rather than summing per-hop timeouts, demonstrated with one downstream artificially slowed.
• A one-page lifecycle map for the service: for each of the seven stops, the metric you watch, the failure it catches, and the fix you applied — the artifact you would hand an on-call engineer.