Capstone: design a complete CI/CD pipeline end to end
A production pipeline is one chain — PR checks gate merge, the artifact is built once with an SBOM and a signed provenance attestation, then the same digest is promoted staging → canary → prod with metric-gated rollback. Skip any link and the chain breaks.
The release looked clean: green PR, green tag build, staging healthy, one approval click, production deploy succeeded. Forty minutes later, p99 latency tripled and the error budget for the quarter was gone in an afternoon. The post-mortem found one line. The production job did its own docker build from the tag instead of pulling the digest that staging had verified — a transitive dependency had published a new patch in the gap, so prod ran a different binary than the one that passed every gate. Every stage worked. The chain didn’t, because one stage rebuilt instead of promoting. There was also no automated rollback, so a human had to notice, page, and revert by hand.
The whole pipeline is one chain, and a chain is only as strong as its skipped link
By the end of this lesson you will be able to name every link in a production-grade pipeline, explain what each one guarantees, and recognise instantly which skipped link caused any given release incident.
Every earlier lesson in this track was a single link: pipelines, testing in CI, delivery strategies, environments and approvals, release automation, supply-chain provenance, scaling. A capstone pipeline is those links welded in order, and the integrative failure mode is the same every time: any link you skip silently breaks the guarantee the whole chain was supposed to give. Rebuild per stage and your “tested artifact” promise is a lie. Deploy with no rollback path and your canary is just a slower outage. Ship an unverified artifact and your SBOM and tests describe a binary that isn’t the one running. The discipline of a senior pipeline is not adding more stages — it is making sure no stage quietly substitutes a weaker guarantee.
There are three load-bearing invariants. (1) PR checks are the merge gate — fast feedback first, required status checks block merge, so main is always releasable. (2) Build once — on merge or tag the artifact is built exactly one time, producing one immutable digest, and from that point nothing is ever rebuilt; it is only promoted. (3) Promote the same digest — staging, canary, and production all deploy the identical image@sha256:…, so the only variable across environments is configuration, never the binary. Hold those three and most release-day surprises become impossible by construction.
Stage 1 — PR checks: fail fast, then prove, then block merge
The PR gate has a shape: cheapest, most-likely-to-fail checks first. Lint and typecheck run in seconds and catch the majority of mistakes, so they go first and short-circuit the expensive work. Then the affected-only build and test — on a monorepo you test what the diff touches, not the world (this is where the scaling lesson pays off). Configure these as required status checks on the branch so a red check physically blocks the merge button; a check that only reports is a suggestion, not a gate. The PR stage never deploys and never signs — its single job is to keep main mergeable-means-releasable.
name: ci
on:
pull_request:
permissions:
contents: read # least privilege: PR checks need nothing more
jobs:
fast-gate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4 pinned by SHA
- uses: actions/setup-node@1d0ff469b7ec7b3cb9d8673fde0c81c44821de2a # v4 pinned by SHA
with: { node-version: '22', cache: 'npm' }
- run: npm ci
- run: npm run lint && npm run typecheck # seconds; fail fast
- run: npx nx affected -t build test # affected-only, not the worldStage 2 — Build once: one digest, SBOM, signed provenance
This is the pivot of the whole pipeline. On merge to main (or on a release tag) you build the image one time and push it by digest. In the same run you generate an SBOM (what is inside), produce a build-provenance attestation (how, where, and from which commit it was built), and sign them with keyless cosign via Sigstore — GitHub’s OIDC token is the signing identity, so there is no long-lived key to leak. This is exactly the chain that earns SLSA Build L2 (and L3 with an isolated, ephemeral reusable-workflow builder): a hosted, hardened builder that generates and signs provenance automatically. Two non-negotiables for a senior workflow: pin third-party actions by full commit SHA (a moved tag is a supply-chain backdoor), and grant least-privilege permissions: per job, adding id-token: write only on the job that actually needs OIDC and attestations: write only where you attest.
build-once:
needs: fast-gate
runs-on: ubuntu-latest
permissions:
contents: read
packages: write # push to GHCR
id-token: write # OIDC for keyless signing
attestations: write # provenance attestation
outputs:
digest: ${{ steps.push.outputs.digest }}
steps:
- uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4
- uses: docker/build-push-action@4f58ea79222b3b9dc2c8bbdd6debcd730a4f6c4f # v6
id: push
with:
push: true
tags: ghcr.io/acme/api:${{ github.sha }}
- uses: actions/attest-build-provenance@e8998f949152b193b063cb0ec769d69d929409be # v1
with:
subject-name: ghcr.io/acme/api
subject-digest: ${{ steps.push.outputs.digest }}
push-to-registry: true
- uses: anchore/sbom-action@d94f46e13c6c62f59525ac9a1e147a99dc0b9bf5 # SPDX SBOM
with: { image: ghcr.io/acme/api@${{ steps.push.outputs.digest }} }Stage 3–4 — Release, then promote the same digest with gates and rollback
Release is bookkeeping on top of the verified artifact: derive the version from conventional commits (semver), tag, and publish the signed image to the registry. Then deployment is pure promotion of one digest. Staging auto-deploys api@${{ digest }} and runs smoke tests. Production is a separate GitHub environment with protection rules — required reviewers and a “no self-approval” setting turn the deploy into a deliberate, audited human gate. Past the gate you do not flip 100% of traffic; you canary: route a small slice, let an automated analysis watch error rate and latency against the baseline for a few minutes, and only widen if the metric gate passes. If an SLO breach trips, the rollout halts and automatically rolls back to the previous digest — the link the war-story pipeline was missing. The same immutable digest flows through every stage; only config changes.
| Stage | Goal | What plugs in here | Failure if skipped |
|---|---|---|---|
| PR checks | Keep main releasable | Lint/typecheck fast gate, affected-only test, required status checks | Broken code merges; every later stage inherits the rot |
| Build once | One immutable digest | SBOM, provenance attestation, cosign keyless, SHA-pinned actions, OIDC | Rebuild per stage → prod runs a different binary than was tested |
| Release | Versioned, published artifact | Conventional commits → semver, tag, push signed image | No traceable version; can’t say what’s deployed or roll forward cleanly |
| Staged deploy | Promote the same digest safely | Environment approval, smoke tests, canary + metric gate, auto-rollback | No rollback path → a bad canary is just a slower, fuller outage |
deploy:
needs: build-once
uses: ./.github/workflows/promote.yml
with:
digest: ${{ needs.build-once.outputs.digest }} # promote, never rebuild
staging:
needs: build-once
environment: staging # auto-deploy
runs-on: ubuntu-latest
steps:
- run: deploy ghcr.io/acme/api@${{ needs.build-once.outputs.digest }}
- run: ./smoke-tests.sh
production:
needs: staging
environment: production # protection rule: required reviewers, no self-approval
runs-on: ubuntu-latest
steps:
- run: deploy-canary ghcr.io/acme/api@${{ needs.build-once.outputs.digest }} --weight 5
- run: analyze-slo --window 5m || { rollback; exit 1; } # metric gate → auto-rollback
- run: promote-to-full --weight 100▸Why this works
Why verify the artifact at deploy time when you already signed it? Because signing at build time and not checking the signature before deploy leaves the gap wide open — anyone who can write to the registry or swap a tag could substitute an image. The production job should cosign verify-attestation (or gh attestation verify) the exact digest against the expected workflow identity before it deploys. Provenance you generate but never verify is decoration; provenance you check at the gate is a control.
How should production receive the artifact that staging already verified?
Staging passed all gates, but production tripled its latency. The prod job ran `docker build` from the tag. What broke?
On the build-once job you add `id-token: write` and pin actions by SHA. Why both, specifically?
Order the stages of the full pipeline from a PR to a fully-rolled-out production release:
- 1 PR checks — fast lint/typecheck gate, affected-only test, required status checks block merge
- 2 Build once — one digest + SBOM + signed provenance attestation (OIDC, SHA-pinned actions)
- 3 Release — semver from conventional commits, tag, publish the signed image
- 4 Staging — auto-deploy the same digest, run smoke tests
- 5 Production gate — manual approval via environment protection rules
- 6 Canary + rollback — small-slice rollout watched by a metric gate; auto-rollback on SLO breach
- 01Walk the full pipeline from PR to production and name what plugs in at each stage and what breaks if you skip it.
- 02Why is 'build once, promote the same digest' the load-bearing invariant, and how does it make the war-story impossible?
A production CI/CD pipeline is a single chain whose guarantee survives only if no link is skipped. PR checks come first as the merge gate: a fast lint/typecheck pass fails cheaply, an affected-only build and test covers what the diff touches, and required status checks physically block the merge so main stays releasable. On merge or tag the artifact is built exactly once into one immutable digest, with an SBOM for what’s inside, a build-provenance attestation for how and where it was built, and keyless cosign signatures using GitHub’s OIDC identity — actions pinned by commit SHA and permissions kept least-privilege per job. Release is bookkeeping on the verified artifact: semver from conventional commits, tag, publish the signed image. Then deployment is pure promotion of that one digest — staging auto-deploys and smoke-tests it, production sits behind an environment protection rule with required reviewers and no self-approval, and past the gate a canary routes a small slice watched by a metric gate that auto-rolls-back on an SLO breach. The integrative failure mode is always a skipped link: rebuild per stage and prod runs an untested binary, deploy with no rollback and a bad canary is a slower outage, ship an unverified artifact and your SBOM describes something else. Build once, verify, and promote the same digest — that is the whole pipeline. Now when you see a release incident, your first question is: which link was skipped? Rebuild instead of promote, no rollback, or an unverified artifact — one of those three is almost always the answer.
Practice
Start at the top. Tasks go easiest → hardest: recall a fact, apply it to a case, then a senior-level stretch. Open one, attempt it, then reveal.
Something unclear?
Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.