Reproducible builds: SOURCE_DATE_EPOCH, digest-pinned bases, and why a churning layer digest breaks signing

What “reproducible” means and why digests churn

If your team signs release images, ask: can you rebuild the exact same digest from the tagged commit today? If you cannot answer yes, the signature you are shipping proves less than you think. Reproducible means: the same source, built anywhere, any day, produces an image with the same layer digests down to the byte. Because everything in OCI is content-addressed, a single differing byte anywhere in a layer tarball changes that layer’s sha256 and every digest above it. Three usual suspects inject those differing bytes:

• Embedded timestamps. BUILD_TIME=$(date), a build-info file with the current time, or a compiler that stamps the build date — any of these writes a value that changes every second, so the layer never reproduces.
• File mtimes in the archive. When COPY . . writes files into a layer, the layer tarball records each file’s modification time. A fresh git checkout sets mtimes to now, so the same source copied on two days produces two different tarballs with identical contents but different timestamps — different digest.
• Unpinned base images. FROM node:22 resolves to whatever digest the tag points at today; the tag moves on every minor patch, so a rebuild months later pulls a different base and inherits its different layers.

The fix for each is direct. Pin the base by digest: FROM node:22@sha256:abc... so the base never moves. Drive timestamps from SOURCE_DATE_EPOCH — a standardized environment variable holding a fixed Unix time (typically the commit’s author date) that build tools honor instead of the wall clock. And have BuildKit normalize the mtimes it writes.

# syntax=docker/dockerfile:1
FROM node:22@sha256:1a2b3c...      # pinned: base never silently moves
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci
COPY . .
RUN npm run build

# Feed a fixed time and ask BuildKit to rewrite layer timestamps to it:
SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct) \
  docker buildx build \
    --output type=image,name=app:v2.4.1,rewrite-timestamp=true .

rewrite-timestamp=true tells BuildKit to set every file’s mtime in the produced layers to SOURCE_DATE_EPOCH, erasing the checkout-time variance. With the base pinned, the clock pinned, and mtimes normalized, two builds of the same commit on two machines produce the same digests.

Determinism beyond the three big levers

Pinning the base, the clock, and mtimes handles most cases, but a few subtler non-determinisms remain worth knowing. Package installs that do not pin versions (apt-get install curl without =version, pip install without a lockfile) pull whatever is current, so freeze versions or use a lockfile. Build steps that emit output in filesystem-iteration order can vary; tools increasingly sort deterministically, but a custom script may need an explicit sort. Embedded build paths (a compiler recording /home/runner/...) differ across machines; many toolchains accept a path-prefix remap. And network access during build is inherently non-reproducible — the same curl https://... can return different bytes tomorrow — which is why hermetic builds fetch only pinned, checksummed inputs. The senior posture: treat every input to the build as something that must be pinned or normalized, and assume any unpinned input will eventually move and break your digest when you least want it to.