Tags move, digests do not: pinning and signing for a verifiable supply chain
Tags move, digests do not: a tag is a mutable label, a digest is the sha256 of the manifest and fixes the bytes. Signing (cosign) binds a claim to the digest, stored as a referring artifact. Deploy by digest and verify the signature, or you trust whatever the tag last pointed at.
The post-incident timeline was the kind nobody wants to write. At 14:02 a developer fixed a typo in a Dockerfile comment, rebuilt, and ran docker push internal/checkout:prod — the same tag the production cluster pinned. They did not know the cluster used :prod directly. The rebuild had pulled a fresh base image overnight; the new base bumped a TLS library to a version with a regression that closed keep-alive connections aggressively. No deploy was triggered, no PR merged, no approval requested — but over the next hour, as nodes recycled pods and re-pulled :prod, half the fleet silently swapped to the broken image while the other half stayed on the old one. Checkout error rate climbed in a staircase nobody could correlate to a change, because in the change log there was no change. The tag :prod had moved out from under a running system. The fix that shipped that night was one rule: production references a digest, internal/checkout@sha256:…, and a signature gate refuses any image whose digest is not signed by the release key.
The digest is the manifest’s hash
Why does pinning to a digest actually guarantee immutability when pinning to a tag does not? The answer is in how a digest is produced. A digest is not assigned, it is computed: sha256:<hex> where the hex is the SHA-256 of the canonical manifest bytes. That has a precise consequence — the digest covers the manifest, and the manifest lists the config digest and every layer digest, each of which covers its own bytes. So a single manifest digest transitively pins the entire image: change any layer, the layer’s digest changes, the manifest’s content changes, the manifest digest changes. internal/checkout@sha256:9f3c… therefore names one specific tree of bytes that can never be anything else. A tag has no such property: internal/checkout:prod is an entry in the registry’s mutable index mapping the string prod to some manifest digest, and a PUT /v2/.../manifests/prod overwrites that mapping. The registry will even tell you the current digest of a tag — a HEAD /v2/<name>/manifests/prod returns the manifest’s digest in the Docker-Content-Digest response header — which is exactly how CI should resolve a tag to a digest at release time and never look at the tag again.
# resolve the tag you just built to its immutable digest, then deploy THAT
digest=$(docker buildx imagetools inspect internal/checkout:prod \
--format '{{.Manifest.Digest}}') # sha256:9f3c…
kubectl set image deploy/checkout app=internal/checkout@${digest}▸Why this works
Why isn’t “just don’t re-push prod” enough? Because the property you need is not a process promise, it is a structural guarantee. A tag policy lives in people’s heads and CI conventions; it fails the moment someone runs docker push by hand, a second pipeline reuses the tag, or a mirror re-tags upstream. A digest reference cannot be wrong even in principle — there is no operation that makes @sha256:9f3c… mean different bytes, so the reference itself enforces what the policy only requested. This is the same reason content-addressing beats names everywhere in the stack: names are claims, hashes are facts.
Signing binds a claim to a digest
A digest fixes which bytes you run, but says nothing about who vouched for them — anyone who can push can create a digest. Signing closes that gap. With cosign, you sign the image’s digest (not its tag — signing a tag would be meaningless since the tag moves), producing a signature plus optional claims (an SBOM, a provenance attestation). In the modern OCI model these are stored back in the same registry as referring artifacts: a signature is itself a small manifest whose subject field is the digest it signs, discoverable through the referrers API (GET /v2/<name>/referrers/<digest>) which lists all artifacts that refer to a given image. Verification is then: resolve tag→digest, fetch the referrers for that digest, and check that a signature from a trusted key (or, with keyless cosign, a trusted OIDC identity recorded in the Rekor transparency log) covers exactly that digest.
cosign sign --key release.key internal/checkout@sha256:9f3c…
cosign verify --key release.pub internal/checkout@sha256:9f3c… # gate: nonzero exit blocks deployThe order is load-bearing: you verify against the digest you are about to run, not against the tag. If you verify :prod and then deploy :prod, a tag move between the two steps means you verified one image and ran another — a time-of-check-to-time-of-use hole. Verifying and deploying the same digest closes it: the thing you checked and the thing you run are bit-identical by construction.
Why this is the supply-chain backbone
Three senior consequences. First, deploy references must be digests — the Hook incident (a hand docker push to :prod silently rolling out a regression with no change record) is structurally impossible when the running reference is @sha256:…, because no push can re-aim a digest. Second, a signature gate must verify the digest you run, not the tag, and ideally enforce it at admission (a Kubernetes admission controller like sigstore policy-controller rejecting unsigned digests) so the guarantee holds even when a human bypasses CI. Third, mutability has a blast radius: a tag move with no audit trail (:latest re-pushed) can roll out an unintended image fleet-wide over the minutes it takes pods to recycle, and because there is no deploy event, your change-correlation tooling is blind — the staircase error graph with an empty change log. Together these three form a chain: digest pinning makes the bytes fixed, signing makes the approval auditable, and admission enforcement makes both hold even under human error. Skip any link and the chain breaks at exactly that point. Pinning plus signing converts “we trust whoever last pushed the tag” into “we run exactly these signed bytes,” which is the entire difference between a name you hope is right and a fact you can prove.
- 01Explain precisely why a digest is immutable and a tag is not, and how CI should turn one into the other.
- 02Why must you verify the signature on the digest you deploy, and what goes wrong if you verify the tag?
Tags move, digests do not — and an entire class of silent production incidents collapses once a team internalizes that. A digest is the SHA-256 of the canonical manifest bytes, and because the manifest enumerates the config and layer digests, a single manifest digest transitively pins every byte of the image; @sha256:… can never name anything else. A tag is the opposite: a mutable entry in the registry index that a PUT to the manifests endpoint overwrites, so :prod or :latest names whatever was last pushed. CI converts a freshly built tag into a stable reference by reading the Docker-Content-Digest header from a HEAD on the manifests endpoint (or imagetools inspect), then deploys that digest and forgets the tag. Signing adds the missing dimension — who vouched: cosign signs a digest (never a tag, which would vouch for nothing fixed), and the signature lands in the same registry as a referring artifact whose subject is that digest, discoverable through the referrers API. Verification must target the digest you are about to run, because verifying a tag and then deploying it re-resolves the tag twice and opens a TOCTOU window a re-push can drive through. The payoff is structural, not procedural: deploying by digest makes a stray docker push to :prod incapable of moving a running rollout, a signature gate (ideally at admission) makes unsigned bytes unrunnable even when a human skips CI, and the staircase error graph with an empty change log — a tag that moved out from under a live system — stops being possible. Names are claims you hope are right; digests and signatures are facts you can prove. Now when you see a staircase error graph with no corresponding change record, the first question to ask is: what tag moved, and which nodes hadn’t yet re-pulled it?
Practice
Start at the top. Tasks go easiest → hardest: recall a fact, apply it to a case, then a senior-level stretch. Open one, attempt it, then reveal.
Something unclear?
Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.