BuildKit as a build graph: LLB, content-addressed cache keys, and why a build is a DAG not a script
BuildKit parses a Dockerfile into LLB — a content-addressed DAG of build operations, not a top-to-bottom script. Independent stages build in parallel, and cache hits are decided by the digests of an operation's inputs, not by line numbers.
CI had been green for months at 45 seconds a build, then one morning every build took six minutes — and nothing in the Dockerfile had changed. The team stared at the same COPY . . line they had shipped for a year, convinced the registry cache was broken. The real change was upstream: someone had added a .env.local to the repo and dropped a generated build-info.txt that CI wrote before the docker build. Every build now produced a slightly different file in the context, so the digest of the COPY . . input changed on every run, every layer after it missed cache, and RUN npm ci reran from scratch — the 5 minutes 15 seconds that used to be one cache hit. The fix was a .dockerignore line, but the lesson was the mental model. BuildKit had not “decided” to rebuild line 8. It had computed a content digest for that operation’s inputs, found no matching cache entry, and correctly executed it. The cache key was never the line number. It was the bytes.
The Dockerfile is a graph, not a script
The classic mental model — “Docker runs my Dockerfile top to bottom, one line one layer” — describes the legacy builder, and it is wrong for BuildKit, the default since Docker 23.0. BuildKit’s frontend parses the Dockerfile into LLB (Low-Level Build definition): a directed acyclic graph of operations. Each node is a vertex like “run this command on this rootfs,” “copy these files from the context,” or “pull this image”; edges are data dependencies. The solver then walks the DAG and executes vertices whose inputs are ready, in parallel, as worker capacity allows.
The practical consequence is parallelism you did not write. Two independent FROM stages — say a Go build stage and a separate stage that fetches static assets — have no edge between them, so BuildKit runs them at the same time on different workers. A serial 6-minute build with two independent 3-minute legs can finish in a little over 3. You get this for free precisely because the build is a graph: the solver sees the legs are independent and schedules them concurrently, where the legacy builder would have run them one after the other.
# syntax=docker/dockerfile:1
# These two stages are independent vertices in the LLB graph —
# BuildKit builds them concurrently, not sequentially.
FROM golang:1.23 AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download # depends only on go.mod/go.sum
COPY . .
RUN go build -o /app ./cmd/server
FROM node:22 AS assets
WORKDIR /web
COPY web/package*.json ./
RUN npm ci # runs in parallel with go mod download
COPY web/ ./
RUN npm run buildCache keys are digests, not line numbers
When you see a step rerun that you were sure would be cached, the instinct is to blame Docker — but the real question is always: what byte changed? Every LLB vertex computes a cache key from the digest of its inputs. For a RUN, the key folds in the digest of the parent layer (the entire filesystem state it runs on top of) plus the command string. For a COPY, the key folds in the parent plus the digest of the exact file contents being copied — computed from the build context, honoring .dockerignore. The solver looks that key up in the local cache and any imported remote cache; a match is a cache hit, and the vertex’s pre-computed output layer is reused without running anything. A miss runs the vertex and stores the result under that key.
This is why ordering instructions correctly is the single highest-leverage build optimization. COPY go.mod go.sum ./ then RUN go mod download before COPY . . means the expensive download vertex depends only on two small files: change application source and those files’ digests are unchanged, so the download is a cache hit while only the cheap go build reruns. Invert it — COPY . . before go mod download — and every source edit changes the copy vertex’s digest, cascading a miss into the download. A correctly ordered Node build hits cache and finishes in ~4 seconds; the inverted one reruns npm ci for ~90 seconds on every one-line change. Same instructions, same final image, 20× difference, decided entirely by which vertex sits upstream of the volatile input.
A Dockerfile has COPY . . on line 8, then RUN npm ci on line 9. A developer edits a single line in src/app.js and rebuilds. Why does npm ci rerun even though package.json did not change?
▸Why this works
Why digest-based keys instead of timestamps or line numbers? Because content addressing makes the cache correct under reordering, sharing, and distribution. A vertex keyed by the digest of its inputs produces the same key on any machine, so a cache populated in CI can be imported on a developer laptop and hit. Timestamp-based invalidation would miss whenever an mtime changed even if bytes did not, and line-based keying would break the moment you moved an instruction. The price is the failure mode in the Hook: anything that perturbs the input bytes — a generated file in the context, an mtime-sensitive build that BuildKit had to read — changes the digest and busts the layer. Content addressing is unforgiving but honest: the cache reflects exactly what went in.
Reading a build log as a graph walk
Once you see the build as a DAG, the log reads differently. CACHED next to a step means the solver found that vertex’s cache key already satisfied. Steps printed interleaved rather than strictly in file order are independent vertices being solved concurrently. A => [build 4/7] and => [assets 2/4] appearing together is the two-stage parallelism, not a rendering glitch. And a step that you expected to be cached but shows as running tells you precisely one thing: the digest of its inputs differs from last time. The debugging question is never “why did Docker decide to rebuild this line” — it is “what input byte changed.” Usually it is the context (add a .dockerignore), sometimes a base image that moved tags (pin by digest), occasionally a build arg folded into the command string.
A Dockerfile defines two stages with FROM that share no COPY --from or other dependency between them. What does BuildKit do that the legacy builder did not?
- 01Explain what LLB is and how BuildKit's cache key for a vertex is computed for a RUN versus a COPY.
- 02Why does instruction ordering produce a 20x build-time difference, and how do you debug an unexpected cache miss?
BuildKit, the default builder since Docker 23.0, does not execute a Dockerfile top to bottom — its frontend compiles the file into LLB, a content-addressed directed acyclic graph whose vertices are build operations and whose edges are data dependencies. The solver runs vertices whose inputs are ready, concurrently wherever no edge forces an order, so two independent FROM stages build in parallel and a serial-looking build finishes in close to the time of its slowest leg. Every vertex carries a cache key computed from the digest of its inputs: a RUN folds in the parent layer’s digest plus the command string, a COPY folds in the parent plus the digest of the exact file bytes copied from the context after .dockerignore filtering. The solver looks that key up in local and any imported remote cache — a hit reuses the stored layer and prints CACHED without executing, a miss runs the vertex and stores its output. This is why line numbers never matter and ordering matters enormously: COPY go.mod and RUN go mod download before COPY . . keeps the expensive download keyed to small stable files, so source edits leave it cached (~4s) and only the cheap compile reruns, while the inverted order busts the install on every change (~90s) — same instructions, ~20x apart. And it is why the mystery six-minute build was never about line 8: a generated file slipped into the context, the COPY vertex’s input digest moved, and everything downstream correctly missed. Now when you see a step marked as running that should have been cached, stop asking “which line is wrong” and start asking “what input byte changed” — the answer is almost always the build context, a moved base tag, or a build arg that snuck into a command string.
Practice
Start at the top. Tasks go easiest → hardest: recall a fact, apply it to a case, then a senior-level stretch. Open one, attempt it, then reveal.
Something unclear?
Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.