Secrets and the supply chain: keeping secrets out of layers, and signing images so they cannot be swapped
A secret in ENV or ARG leaks via docker inspect and bakes into a layer; build secret mounts keep it out. An unsigned image can be swapped at the registry; cosign signs the digest, records it in Rekor, and an admission gate refuses unsigned images — the SLSA build-to-admit chain.
Two failures, one supply chain. The first was quiet: a Dockerfile passed an npm registry token as a build ARG to install a private package, and a year later a security researcher pulled the public image, ran docker history and docker inspect, and read the token straight out of the image metadata — it had been baked into a layer and the inspect output, immutable and shipped to anyone who could pull. The second was louder: an attacker who had compromised a CI service account pushed a tampered image to the same repository under the latest tag, swapping the legitimate binary for one with a backdoor. The cluster pulled latest on the next deploy and ran it without a second thought, because nothing in the path from registry to running container ever asked “is this the image we actually built?” Both holes close the same way — secrets that never enter a layer, and images that carry a cryptographic signature an admission gate verifies before anything runs.
A secret in a layer is a published secret
When you pull any public image, you can read its full history and metadata. That is exactly what an attacker does — and if your team ever passed a credential through a build, knowing that fact changes how you think about every image you have shipped.
The image is an append-only stack of layers, and its metadata is world-readable to anyone who can pull it. That makes every common way of passing a secret into a build a leak. An ENV API_KEY=... line is visible in docker inspect and docker history and is inherited by every process in the container — and by docker inspect from anyone with registry access. A build ARG SECRET is no better: even though it is not in the final environment, build args are recorded in the image history, and any file the build writes the secret into becomes a layer that persists even if a later layer deletes it, because layers are additive — rm-ing a file in layer 5 does not remove it from layer 3, it just hides it, and docker save or a layer-extraction tool reads it right back. Copying a credentials file in, using it, and deleting it is the canonical mistake: the secret is in the image forever, one tar extraction away.
The fix is to give the build the secret without writing it into any layer. BuildKit secret mounts do exactly this: RUN --mount=type=secret,id=npmtoken exposes the secret as a file under /run/secrets for the duration of that one RUN step only, mounted in a tmpfs that is never committed to a layer and never appears in history or inspect. The secret is present while the command runs and gone the instant it finishes, leaving no trace in the image. The same principle governs runtime: never bake credentials into the image at all — inject them at run time from the orchestrator’s secret store (Docker/Kubernetes secrets, a vault sidecar, or mounted files), so the image is identical across environments and carries nothing sensitive. The rule is blunt: if a secret is ever in a layer, ENV, or build ARG, treat it as published and rotate it.
A Dockerfile does COPY creds.json ., uses it during npm install, then RUN rm creds.json as the next step. Is the secret safe in the published image?
▸Why this works
Why does a build secret mount solve what ENV and ARG cannot? Because it changes where the secret lives during the build. ENV and ARG make the secret part of the image’s declarative definition — recorded, inherited, inspectable — and any file written from them is committed to an additive layer. A secret mount makes the secret a transient tmpfs file scoped to one RUN, visible to that command’s processes and unmounted before the layer is committed, so it is never part of the layer’s content or the image’s metadata. The image you publish is byte-identical whether or not a secret was used to build it. That is the property you want: the artifact carries nothing sensitive, so leaking the artifact leaks nothing.
Signing the digest: stopping the swap
Keeping secrets out of the image protects what is in the image; signing protects which image runs. A tag like latest is a mutable pointer — anyone who can push to the repository can repoint it at a different digest, and a tampered image swapped under the same tag pulls and runs exactly like the original. The cryptographic anchor is the image digest (a SHA-256 over the manifest): it is content-addressed and immutable, so a different image has a different digest. Cosign (from the sigstore project) signs that digest and stores the signature as an artifact alongside the image in the registry. Crucially, it can do this keylessly: it gets a short-lived certificate tied to an OIDC identity (your CI’s workload identity) and records the signing event in Rekor, a public append-only transparency log — so there is a tamper-evident, queryable record that “this digest was signed by this identity at this time,” with no long-lived private key to steal.
Signing only matters if something verifies it before running the image, which is the job of an admission controller (Kubernetes admission webhook, OPA/Gatekeeper, or the cosign policy controller): at deploy time it resolves the image to its digest, checks for a valid cosign signature from an allowed identity, and refuses to admit anything unsigned or signed by the wrong key. That single gate defeats the tag-swap: the backdoored image has no valid signature from the trusted CI identity, so it is rejected before a pod ever starts. This is the spine of SLSA (Supply-chain Levels for Software Artifacts): a verifiable chain of provenance from a trusted build, through a recorded signature and transparency-log entry, to an admission gate that only runs artifacts whose origin it can prove. Pin to digests, sign at build, verify at admit — and the registry stops being a place an attacker can quietly substitute your binary.
A team signs every image with cosign and records each signature in Rekor, but deploys still pull by mutable tag with no admission check. An attacker pushes a backdoored image to the same tag. What happens?
- 01Why does COPY-then-rm or a build ARG leak a secret, and what does a BuildKit secret mount do differently?
- 02Walk through how cosign, Rekor, and an admission controller stop a tag-swap attack, and why each piece is necessary.
A container image is an append-only stack of immutable layers whose metadata is readable by anyone who can pull it, so every casual way of passing a secret into a build is a leak. ENV is inherited and inspectable; a build ARG is recorded in history; and any file a build writes a secret into becomes a layer that survives even a later rm, because layers are additive — the deletion is a whiteout that hides the file in the merged view while docker save recovers it intact, which makes COPY-then-rm the textbook mistake. The fix is to keep the secret out of every layer: a BuildKit secret mount exposes it as a transient tmpfs file scoped to one RUN, never committed and never in history, leaving the published image byte-identical whether or not a secret was used; at runtime, inject credentials from the orchestrator’s secret store rather than baking them in. If a secret ever touched a layer, ENV, or ARG, treat it as published and rotate it. Keeping secrets out protects what is in the image; signing protects which image runs. A mutable tag like latest lets anyone with push access repoint it at a tampered digest, so the anchor is the immutable content-addressed digest: cosign signs that digest keylessly via a short-lived OIDC-bound certificate and records the event in the Rekor transparency log, and an admission controller verifies the signature against an allowed identity before admitting the image, rejecting anything unsigned or wrongly signed — defeating the tag-swap before a pod starts. Together, pin to digests, sign at build, and verify at admit, this is the SLSA chain: a verifiable path from a trusted build through a recorded signature to a gate that runs only artifacts whose origin it can prove. Now when you see a Dockerfile that passes a token through ARG, or a Kubernetes deployment that pulls by mutable tag with no admission check, you know exactly what to fix and why each step in the chain is non-optional.
Practice
Start at the top. Tasks go easiest → hardest: recall a fact, apply it to a case, then a senior-level stretch. Open one, attempt it, then reveal.
Something unclear?
Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.