open atlas
↑ Back to track
Deployment & Infra DEP · 09 · 03

Image security and scanning

An image ships its whole filesystem as attack surface: minimise the base, drop to non-root, pin by digest, and scan in CI and over time — because a base CVE that was clean at build can turn critical while it sits in prod.

DEP Senior ◷ 17 min
Level
FoundationsJuniorMiddleSenior

The pentest report lands with one line in red: remote code execution, CVSS 9.8, in a TLS library nobody on the team has ever imported. It came in with the base image — node:18, pulled clean eleven months ago, scanned green at build, deployed, and never looked at again. The CVE was published four months after the build. The image never changed; the threat landscape did. Nothing in the pipeline was watching the already-shipped image, so a critical flaw sat in production for a third of a year with a green checkmark next to it.

By the end of this lesson you will know exactly which four decisions harden an image, why a green CI scan is not a security guarantee, and what a production pipeline must do that a build pipeline cannot.

Attack surface is what you ship

Every byte in your image is something an attacker can use once they are inside the container. A full node:18 or python:3.12 base is a complete Debian or Ubuntu userland: a shell, apt, curl, bash, dozens of system libraries, a package manager — hundreds of packages, each a row in a CVE database. You did not ask for any of it; it came with the convenience of FROM node:18. The first and largest lever in image security is therefore not a scanner or a policy — it is carrying less.

Three moves shrink the surface. Slim variants (node:22-slim) strip the base OS down to roughly what the runtime needs. Distroless images (gcr.io/distroless/*) go further: only your app plus its direct runtime libraries — no shell, no package manager, no apt. Multi-stage builds ensure the compilers, dev dependencies, and apt caches used to build the artifact never reach the final stage at all. The payoff is direct: fewer installed packages means fewer CVEs to triage, a smaller image to pull and store, and far less for an attacker who lands a shell — because on distroless there is no shell to land in.

# syntax=docker/dockerfile:1
FROM node:22 AS builder
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci
COPY . .
RUN npm run build

# Distroless final stage: app + runtime libs only, no shell, no apt
FROM gcr.io/distroless/nodejs22-debian12
WORKDIR /app
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/node_modules ./node_modules
# Distroless ships a nonroot user; do not run as root
USER nonroot
CMD ["dist/server.js"]

The three base choices trade attack surface against convenience:

BaseTypical sizePackages / CVE surfaceShell + pkg manager?Debuggability
node:22 (full)~1.1GBHundreds — largeYes (bash, apt)Easy — full userland
node:22-slim~250MBDozens — moderateYes (smaller set)Workable — has a shell
distroless/nodejs22~120MBApp + runtime only — minimalNoHard — logs + ephemeral debug containers

Run as non-root, and pin what you run

By default a container process runs as root inside its namespace. That root is somewhat constrained, but it is still the wrong default: if an attacker breaks into the process — through an app bug, a dependency RCE, a misconfigured mount — they inherit root, and root plus a kernel bug or a careless --privileged flag or a writable host bind mount is a path to escalation off the container. Docker’s own guidance is blunt: if a service can run without privileges, use USER to change to a non-root user. Add an explicit unprivileged user and switch to it before CMD. Pair it with a read-only root filesystem (--read-only, with a tmpfs for the few paths that genuinely need writes) so a compromised process cannot drop a binary or rewrite your app on disk.

# Create and switch to an unprivileged user before CMD
RUN addgroup --system app && adduser --system --ingroup app app
USER app
# At run time, lock the root filesystem read-only:
#   docker run --read-only --tmpfs /tmp myimage

The second discipline is pinning by digest, not tag. A tag like node:22 is a moving pointer — the publisher can, and does, repoint it to a new image whenever they rebuild. That means FROM node:22 is not reproducible: two builds a week apart can pull two different base images, and :latest is the worst of all, silently sliding under you. A digest (node:22@sha256:...) is the immutable content hash of one exact image; pin to it and every build, every machine, every CI run gets byte-identical bytes — and you have a fixed target to scan and to audit. The cost is that you must deliberately bump the digest to get upstream patches, which is exactly why scanning over time (below) matters.

Why this works

“Pin the digest” and “get security patches” seem to fight: an immutable digest never receives the upstream fix. They don’t, if you separate the two jobs. The digest gives you a known, reproducible input — you always know precisely what is in prod. Continuous rescanning gives you the signal to move — when a CVE lands in your pinned digest, the scanner flags it and you bump to a new, patched digest deliberately, through a PR and CI, instead of a tag silently mutating under you. Dependabot/Renovate automate the bump. You get reproducibility and patching, just not by accident.

Scanning: at build, and forever after

A scanner (Docker Scout, Trivy, Grype) builds a software bill of materials — the SBOM, a complete inventory of every OS package and language dependency in the image, with versions — and matches that inventory against a continuously updated vulnerability database to pinpoint CVEs in your layers and packages. Wire it into CI as a gate: fail the build on new critical/high findings so a vulnerable image never ships. The SBOM is also your provenance and audit record — when the next big CVE drops, you answer “are we affected?” by querying SBOMs, not by guessing.

# Inventory + vulnerability summary (Docker Scout)
docker scout quickview my-app:1.4.0
docker scout cves my-app:1.4.0

# Generate an SBOM for audit / provenance
docker scout sbom my-app:1.4.0

# Equivalent with Trivy, gating CI on critical/high
trivy image --severity CRITICAL,HIGH --exit-code 1 my-app:1.4.0

The senior failure mode is treating the build-time scan as the whole job. A green scan is a snapshot, not a guarantee. New CVEs are disclosed daily against packages that are already inside your shipped image; an image clean at build can be critical a month later with zero code changes — exactly the hook. So scanning must run on a schedule against deployed images, not only in the build that produced them, and the alert must route to whoever can ship the patched digest. Rescan registry images nightly; treat “no one re-scanned” as the real vulnerability.

Pick the best fit

A compiled service must ship hardened to production with the smallest defensible attack surface. Pick the final-stage base.

Quiz

An image scanned green at build is found a month later with a critical CVE, with no code or image changes in between. What happened, and what prevents it?

Quiz

Why prefer FROM node:22@sha256:... over FROM node:22 for a production base image?

Recall before you leave
  1. 01
    Why does minimising the base image (slim/distroless + multi-stage) improve security, not just size?
  2. 02
    Walk through how a base-image CVE can sit in production for months with a green checkmark, and the practices that prevent it.
Recap

An image ships its entire filesystem as attack surface, so the first and biggest security lever is carrying less: choose a minimal base (slim, or distroless with no shell or package manager) and use multi-stage builds so compilers and dev dependencies never reach the final image, which directly shrinks how many CVEs you must triage and how much an intruder can pivot with. Then harden what remains — add an explicit non-root USER before CMD so a compromised process doesn’t inherit root, and lock the root filesystem read-only with a tmpfs for the few writable paths. Pin the base by digest (@sha256:…) rather than a mutable tag so builds are reproducible, auditable, and present a fixed target, accepting that you then bump the digest deliberately to take patches. Finally, scan with a tool that builds an SBOM and matches it against a CVE database — gate CI on critical/high to keep vulnerable images from shipping, but also rescan deployed images on a schedule, because a base CVE that was clean at build can turn critical while the unchanged image sits in production, and the real vulnerability is no one ever looking again. Now when you see a “green” build scan badge on an image that’s been in production for six months, ask yourself: when did that pipeline last check the running image, not the one that built it?

Practice

Start at the top. Tasks go easiest → hardest: recall a fact, apply it to a case, then a senior-level stretch. Open one, attempt it, then reveal.

recallapplystretch0 of 5 done

Something unclear?

Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.

shortcuts expand
search
K
prev piece
k
next piece
j
cycle tier
t
this menu
?
sources2
expand
  1. 01
  2. 02

Trademarks belong to their respective owners. Editorial reference only.