Deployment & Infra
Deployment capstone: ship a production pipeline
You have built each stage of the chain in isolation. The capstone is to compose them into one release that survives a deploy under live traffic — and to prove it by deliberately driving each seam to the edge and showing the contract holds. This is the artifact you would point a new hire at to explain how your service actually ships.
Turn the whole track into one reproducible pipeline: build an immutable image, declare the k8s objects, pick and configure a rollout strategy with real health gates, inject secrets at runtime, front it with an L7 load balancer that drains, and codify every bit of it as IaC — then demonstrate a zero-downtime release including a schema change and a rollback.
Design and ship a complete production deployment pipeline for a small stateful HTTP service (an API backed by a relational database), and prove it survives a zero-downtime release — including a database migration and a rollback — with traffic flowing the whole time.
- A single repo (or directory) where terraform/helm apply from clean recreates the entire running service — image digest, objects, probes, secret wiring, LB, and rollout config — with zero manual kubectl edits.
- A release trace under live load showing error rate staying at baseline and p99 latency not spiking across the cutover; the readiness probe is shown failing during a pod's startup and passing only once the DB pool is open.
- Evidence the schema change shipped via expand-contract: the three deploys (expand, migrate/dual-write, contract) are distinct, and a rollback executed between them serves correctly because no intermediate schema broke N-1 compatibility.
- Proof that no secret is in the image (docker history is clean) and that secrets resolve only at runtime; a note on how the k8s Secret is protected beyond base64 (encryption-at-rest or external manager).
- A one-page architecture write-up naming each stage, the contract at each seam, and which configuration enforces it — the document a new hire reads to understand how the service ships.
- Add a CI/CD pipeline that builds the image, scans it for CVEs, runs an IaC plan, gates the merge on a clean drift check against the live cluster, and promotes to prod only after a canary clears its metric gate automatically.
- Add automated drift detection: a scheduled IaC plan that alerts on any diff between declared and actual state, and a policy/admission check that rejects out-of-band kubectl edits.
- Run a fault drill: kill a pod mid-request, expire the readiness dependency, and apply a deliberately incompatible migration on a branch — show that the drain, the probe, and the expand-contract discipline each prevent a user-visible outage, and capture the signals that caught the bad migration before it reached prod.
- Implement progressive delivery with an automated canary controller (Argo Rollouts / Flagger) that shifts traffic in steps, watches golden signals, and auto-rolls-back on a metric breach — then trigger an auto-rollback and capture the evidence.
This is the loop every real release runs: build an immutable artifact, declare the objects, make “ready” mean serving with a real probe, pick a rollout strategy whose success is metric-gated, keep the schema N-1 compatible with expand-contract so the rollout stays reversible, drain in-flight requests at L7 with SIGTERM cooperation, inject secrets at runtime and protect them beyond base64, and codify the whole thing as IaC so the environment you tested is the one you shipped. Composing the stages — and engineering every seam between them — once on a toy service is what makes the production version muscle memory.