Container image security

An image is inherited layers, not your code

A container image is not your application. It is a stack of read-only filesystem layers — a base OS layer, a language runtime, system libraries, then your code and dependencies on top — packaged as a tarball with a JSON manifest, addressed by a content hash (the digest, a sha256:…). When you write FROM node:18, you are inheriting hundreds of megabytes of other people’s binaries: a Debian userland, OpenSSL, glibc, the Node runtime, and every package those pull in. Your COPY . . is the thin top layer. The security-relevant truth is that most of the attack surface in a typical image is below your line — in layers you didn’t write and rarely audit.

That inheritance is the first problem, because vulnerabilities live in those lower layers. A 2023-era survey of public images routinely found dozens to hundreds of known CVEs per image, the overwhelming majority in OS packages and language base layers, not application code. A critical OpenSSL or glibc CVE lands in your image the moment you build on a base that contains the vulnerable version — you shipped it without writing a line of vulnerable code. So “is my image secure?” is mostly “what did I inherit, and do I know its current CVE state?”

Tags float; digests pin

The second problem is identity. A tag like node:18 or myapp:latest is a mutable pointer: it names a manifest today, but the registry owner (upstream, or anyone with push access) can re-point it tomorrow. FROM node:18 is not “this exact image” — it’s “whatever node:18 resolves to at pull time.” Two builds a week apart can produce materially different images from the identical Dockerfile, which is why “it built clean last week” tells you nothing about what you’re running now.

The fix is to address images the way the registry actually stores them: by digest. FROM node:18@sha256:abc123… pins the exact content-addressed bytes; the digest is the hash of the manifest, so a different image is a different digest, period. Pin base images by digest in your Dockerfile, and pin your deployed image by digest in your Kubernetes manifest (image: myapp@sha256:…, not image: myapp:latest). This makes deploys reproducible and closes the re-push attack: even if someone re-points the tag at a poisoned manifest, your digest-pinned reference won’t follow it. The tradeoff is real — digests are opaque and don’t auto-update, so you trade convenience for control and need a bot (Renovate/Dependabot) to bump the pinned digest deliberately, which is exactly the point: updates become decisions, not surprises.

Minimal base: you can’t be exploited through a shell you don’t ship

The third control attacks the inherited surface directly: use the smallest base that runs your app. A full ubuntu or debian base ships a shell, a package manager, curl/wget, coreutils, and hundreds of libraries — every one a potential CVE and every one a post-exploitation convenience. The progression is debian (≈120 MB, full userland) → debian-slim (≈75 MB, trimmed) → alpine (≈5–8 MB, musl libc + busybox) → distroless (Google’s images: just your app, its runtime deps, CA certs, and /etc/passwd — no shell, no package manager, no busybox) → scratch (literally empty; for a static binary).

The payoff is double. First, fewer packages means fewer CVEs to track and patch — a distroless or scratch image can carry single-digit or zero OS CVEs versus dozens in a full base, which directly shrinks the scan-and-rebuild treadmill. Second, and underrated: with no shell in the image, an attacker who achieves RCE can’t sh -c, can’t curl | sh a second-stage payload, and can’t use the usual living-off-the-land binaries — they’re reduced to whatever their single exploit primitive gives them. The cost is operational: you can’t kubectl exec a debug shell into a distroless pod (you use ephemeral debug containers instead), and a musl-based Alpine can surface subtle libc differences (DNS resolution, some glibc-only binaries). For most services the tradeoff favors minimal; reach for a fuller base only when you’ve hit a concrete compatibility wall.

Scan and rebuild: a CVE-free image rots in place

Scanning is the control everyone knows and the one most often misunderstood. A scanner (Trivy, Grype, Clair, or your registry’s built-in) reads the image’s package metadata, matches installed versions against vulnerability databases (the NVD, distro security trackers, GitHub Advisory), and reports CVEs by severity. The senior insight is twofold. First, scan at build time and fail the pipeline on new criticals — a scan that only runs in a dashboard nobody watches is theater; the value is the gate that stops a vulnerable image from being pushed at all. Second, and more subtle: an image’s CVE count goes up over time even though the image never changes, because new CVEs get disclosed against packages already inside it. The image you scanned clean in January can be carrying three new criticals in March without a single byte changing. So the real control isn’t “scan once”; it’s a loop: rescan deployed images on a schedule against the current database, and rebuild on a fresh base to absorb upstream patches. Rebuilding is how the fix actually lands — the patched OpenSSL only enters your image when you rebuild on a base that contains it.

This is also why minimal bases and scanning reinforce each other: the fewer packages you inherit, the fewer CVEs get disclosed against you later, so the rebuild treadmill spins slower.

Run non-root and prove provenance

Two more controls close the loop. Run as a non-root user: by default a container’s process runs as root (uid 0) inside the container, and while that’s not the host’s root, it’s a needless escalation — set a USER directive in the Dockerfile (or runAsNonRoot: true in the pod) so a process bug lands as an unprivileged user, shrinking the blast radius of anything that goes wrong. It’s free for almost every app and required by the restricted Pod Security Standard. (Runtime hardening — capabilities, seccomp, read-only root — is the next lesson; here the build-time slice is just: don’t bake root in.)

Prove provenance with signing. Pinning by digest guarantees you run a specific image; signing guarantees that image came from you. With Sigstore’s cosign you sign the image (keylessly, via OIDC, so there’s no long-lived key to leak) and attach attestations — an SBOM (the software bill of materials: the full dependency inventory, so when the next Log4Shell drops you can answer “are we affected?” in minutes instead of days) and a SLSA provenance statement (who built it, from which source commit, on which builder). Then an admission controller in the cluster (Kyverno, Sigstore Policy Controller, Connaisseur) verifies the signature before the pod is allowed to run and rejects anything unsigned or signed by the wrong identity. That is the link that defeats a registry compromise: even if an attacker pushes a malicious image, it isn’t signed by your build identity, so admission refuses it.