Verifying provenance and policy gates
L01 signed images and produced SLSA provenance — but a signature nobody checks protects nothing. The enforcement half: verify signature + provenance + builder/source identity at the deploy boundary via an admission controller, fail closed, as policy-as-code.
The team had done everything right in CI. Every image was signed with cosign, every build emitted a SLSA provenance attestation, the green checkmarks were beautiful. Leadership got a slide that said “supply chain: secured.” Then, during a Friday firefight, an engineer needed a hotfix out fast, built an image on their laptop, and ran kubectl set image straight at the prod cluster. It ran. No signature. No provenance. Nothing said a word — because nothing in the cluster ever checked. The signatures were decoration. The whole pipeline that produced cryptographic proof of origin was hanging proof on the wall while the door stood open. This lesson is the half that was missing: not how to sign, but how to make the cluster refuse anything that isn’t signed and provenanced by a builder you trust — and to refuse it even when verification itself goes wrong.
Signing is a claim; verification is the protection
L01 gave you the produce half: cosign sign attached a signature to an image digest, and the build platform emitted a SLSA provenance attestation describing what built it. That is real cryptography. It is also, on its own, completely inert. A signature is just a claim bolted onto an artifact — it changes nothing about what runs. The security only exists at the moment something refuses to run an artifact that isn’t signed and attested by a trusted builder. If nothing refuses, the signatures are wall art.
So the question this lesson answers is not “how do I sign” — you did that — but “what, exactly, do I verify, and where does the refusal live.”
What you verify: signature, provenance, and identity — by digest
Verification is three checks, and skipping any one of them leaves a hole:
# 1. SIGNATURE — is this exact image digest signed by an identity I trust?
# Keyless: the signer's OIDC identity is recorded in Rekor (the transparency log)
# and the cert chains to Fulcio. You pin WHO, not a long-lived key.
cosign verify \
--certificate-identity "https://github.com/acme/app/.github/workflows/release.yml@refs/heads/main" \
--certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
ghcr.io/acme/app@sha256:abc123...
# 2. PROVENANCE — does a SLSA attestation say MY builder built it from MY source commit?
cosign verify-attestation \
--type slsaprovenance \
--certificate-identity "https://github.com/acme/app/.github/workflows/release.yml@refs/heads/main" \
--certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
ghcr.io/acme/app@sha256:abc123...The first check answers “is this signed by someone I trust?” The second answers a stronger question: “was this image built by my CI, from my repository, at a commit I can name?” — which is why provenance matters even for a “signed” image. An attacker who steals a signing key can sign a malicious image; they cannot easily forge a provenance attestation claiming your build platform produced it from your source. And every check is pinned to the image digest (@sha256:...), never a mutable tag — a tag can be repointed at a different image after you verified it; a digest is the content.
The third check: a policy on the attestation
Verifying the signature is valid is not enough — you must assert the contents match your expectations. Policy-as-code expresses this so it is version-controlled, testable, and applied uniformly:
# OPA/Rego sketch: only admit if builder == our CI AND source repo == ours
package supplychain
default allow = false
allow {
input.provenance.builder.id == "https://github.com/acme/app/.github/workflows/release.yml@refs/heads/main"
input.provenance.invocation.configSource.uri == "git+https://github.com/acme/app@refs/heads/main"
input.image.digest == input.requested.digest
}This is the difference between “a valid signature exists” and “a valid signature from my builder for my source on this exact digest.” A leaked key, a fork, a rebuild from a tampered branch — all produce signatures that are cryptographically valid and policy-rejected.
Where the refusal lives: admission control
You could run cosign verify in your CD deploy job. It is better than nothing, and it is a weak gate: anyone who deploys around CI — an operator with kubectl, a compromised CD step, a break-glass script — bypasses it entirely. That is exactly the Friday-hotfix incident. The CI-side check protects the path that goes through CI, which is not the path attackers and panicking engineers use.
The enforcement that nothing bypasses lives at the runtime boundary: a Kubernetes admission controller. On every Pod create, the controller (Sigstore’s policy-controller, Kyverno, or OPA Gatekeeper) intercepts the request, resolves the image to its digest, verifies signature + provenance against your policy, and denies admission if it fails. There is no “deploy around it” — the only way to run a Pod is through the API server, and the API server calls the webhook.
# Sigstore policy-controller: only signed + provenanced images from our CI may run
apiVersion: policy.sigstore.dev/v1beta1
kind: ClusterImagePolicy
metadata:
name: require-acme-provenance
spec:
images:
- glob: "ghcr.io/acme/**"
authorities:
- keyless:
identities:
- issuer: "https://token.actions.githubusercontent.com"
subject: "https://github.com/acme/app/.github/workflows/release.yml@refs/heads/main"
attestations:
- name: must-have-slsa-provenance
predicateType: "https://slsa.dev/provenance/v1"
# CRITICAL: deny when verification cannot complete, do not admit on error
policy:
fetchConfigFile: trueSLSA levels: the bar you verify against
SLSA is the framework that defines how strong the provenance guarantee is. Higher levels mean the provenance was generated by the build platform itself (not the build script), builds are isolated and non-falsifiable, and the path from source to artifact is tamper-resistant. You pick a target level — say, “all prod images must meet SLSA Build L3” — and your policy verifies that incoming artifacts carry provenance meeting that bar before they deploy. The level is the contract; admission control is what enforces it.
Fail-closed vs fail-open: the tradeoff that decides everything
Here is the subtlety that turned the second half of the war story. A verification check can fail to complete: the registry is down, the transparency log is briefly unreachable, the attestation is missing. The policy must decide what to do then, and there are only two answers:
- Fail closed — if verification can’t complete, deny. The artifact does not run.
- Fail open — if verification can’t complete, admit anyway. The artifact runs unverified.
A fail-open policy is silently no protection: every transient outage of the signing or log infrastructure becomes a window where anything ships. The team in the war story shipped a fail-open policy first; when Rekor was briefly unreachable, unverified images sailed straight through, and nobody noticed because there was no denial to alert on. Fail closed is the only correct default.
The cost is real and you design for it, not around it: fail-closed means a registry or log outage can block deploys. You buy availability back by caching trust roots, monitoring the verification path as a first-class dependency, and treating signing infrastructure as production-critical — not by weakening the policy to fail-open, which trades a deploy-availability problem for a security non-feature.
▸Why this works
Why is signing in CI worthless without an admission policy that verifies at deploy? Because a signature is only a claim attached to an artifact; it changes nothing about what runs unless a verifier actively refuses unsigned or untrusted artifacts. If the only check is optional, or lives CI-side, then anyone who deploys around CI — an operator with kubectl, a compromised CD job, a break-glass hotfix — ships unverified code, and the signatures never get looked at. Enforcement has to live at the runtime admission boundary, where the act of running an artifact is gated, not the act of building it. And it must fail closed: if it admits on verification error, every outage of the signing infra silently reopens the door it was supposed to lock.
You must guarantee that only trusted, provenanced images can run in your cluster — even against an operator deploying around CI or a compromised CD step. Which control delivers that?
Your CI signs every image and emits SLSA provenance, and all builds are green. Why does this, on its own, not protect production?
An unsigned image reached prod even though CI signed everything. What enforcement was missing, and what failure mode must that enforcement specifically avoid?
- 01L01 already signs images and emits SLSA provenance. Explain precisely why that does not protect production on its own, what three things you must verify, and where the verification has to live.
- 02Walk through admission control as the enforcement point, the role of SLSA levels, and why the policy must fail closed rather than fail open.
L01 produced cryptographic proof of origin — cosign signatures and SLSA provenance — but proof nobody checks protects nothing; this lesson closes the loop by verifying and enforcing at the deploy boundary. A signature is only a claim bolted onto an artifact: security exists exactly when something refuses to run an artifact that isn’t signed and provenanced by a trusted builder. Verification is three checks, all pinned to the immutable image digest: the signature is from a trusted identity (keyless via Fulcio, recorded in the Rekor transparency log); a SLSA provenance attestation says your builder built it from your source commit (so a stolen signing key, which can sign malice, still fails because it can’t forge your build platform’s provenance); and a policy-as-code rule asserts builder id == your CI, source repo == yours, digest matches. SLSA levels are the bar — higher levels mean platform-generated, isolated, non-falsifiable builds — and you verify incoming artifacts meet your target level before deploy. The refusal must live at the runtime admission boundary: a Kubernetes admission controller (policy-controller, Kyverno, Gatekeeper) intercepts every Pod create and denies on failure, the backstop nothing bypasses — whereas a CI-side cosign verify only guards the CI path and is bypassed by an operator with kubectl, a compromised CD step, or a break-glass hotfix (the war story: an unsigned laptop build ran in prod because nothing checked, then was denied once an admission policy existed). Finally the policy must fail CLOSED: if verification can’t complete because the registry or transparency log is unreachable, it denies rather than admits, because fail-open is silently no protection — every signing-infra outage becomes a window where unverified images ship (the war story’s second bug, a brief Rekor outage). You buy availability back by caching trust roots and monitoring the verification path, never by weakening to fail-open. Now when you see “all images signed — supply chain secured” on a slide, the question to ask is: what exactly refuses to run an unsigned or unprovenanced image at the cluster boundary, and does it fail closed?
Practice
Start at the top. Tasks go easiest → hardest: recall a fact, apply it to a case, then a senior-level stretch. Open one, attempt it, then reveal.
Something unclear?
Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.