Layers and content-addressing: diff_id vs descriptor digest, chain_id, and the gzip non-determinism trap
A layer has two sha256 identities — the diff_id over the uncompressed tar (drives chain_id and overlay dedup) and the descriptor digest over the compressed blob (drives pull). Confuse them and gzip non-determinism stores identical layers twice and shreds cross-runner cache.
The CI cache hit rate fell off a cliff — 92% down to 31% — the week the team moved builders from one cloud to another. Nothing in the Dockerfiles had changed. The vendor/ layer was byte-identical: same go.mod, same go.sum, same source. Yet every build re-pushed a fresh ~180 MB dependency layer and every downstream image saw a cache miss on it. The registry, which had been storing that layer once, now held two copies under two different digests. The difference was invisible until someone unpacked both layers and diffed the tar inside: identical, every byte. The difference was in the gzip wrapper. The old builders ran one zlib version at compression level 6; the new managed builders ran a different zlib at level 9. Same bytes in, different compressed bytes out, different sha256 over the compressed blob — and the registry content-addresses on exactly that compressed digest. Two layers the daemon would happily have shared on disk were, to the registry, two unrelated blobs. The fix was pinning the compressor in the build image; the lesson was that a layer has more than one identity, and you have to know which one each subsystem keys on.
By the end of this lesson you will know which hash each subsystem keys on — and that knowledge is the difference between a 92% cache-hit rate and a 31% one.
One layer, two hashes
When you hit a mysterious cache miss or an unexpected registry storage cost, ask yourself: which hash is the relevant subsystem actually keying on? A layer is a tar archive of filesystem changes. It gets hashed twice, at two different stages, and the two hashes serve different subsystems:
- The diff_id is
sha256of the uncompressed tar. It is what appears in the config’srootfs.diff_ids, and it is the identity the runtime cares about — overlay2 keys its on-disk layer store and its stacking on diff_ids. - The descriptor digest is
sha256of the compressed blob (thetar+gzipyou actually transfer). It is what appears in the manifest’slayersarray, and it is the identity the registry and pull path care about — the URL of a blob is its compressed digest.
$ docker inspect --format '{{json .RootFS.Layers}}' myapp:1 | jq .
[
"sha256:1f3e...", # diff_id of base layer (uncompressed)
"sha256:8a2c...", # diff_id of vendor layer
"sha256:b91d..." # diff_id of app layer
]
# but the manifest's layer digests are DIFFERENT hashes — over the gzipped blobs:
$ docker manifest inspect myapp:1 | jq '.layers[].digest'
"sha256:9d77..." # descriptor digest of base (compressed)
"sha256:c4e0..." # descriptor digest of vendor (compressed)So one physical layer carries two sha256 values. Pull a blob, decompress it, and the daemon verifies the result against the diff_id; the registry only ever saw the descriptor digest. This is normally invisible — until the compressed digest moves while the uncompressed one does not, which is exactly the Hook.
Two builders produce a byte-identical tar for the vendor layer but gzip it with different zlib versions. What does the registry see, and does the runtime still dedup it on disk after pull?
▸Why this works
Why hash twice instead of once? Because compression is a transport concern and the filesystem is a runtime concern, and they want different guarantees. The registry needs to address and verify exactly the bytes it stores and ships, which are compressed — so it hashes those. The runtime needs to verify and stack the actual filesystem content regardless of how it was shipped — so it hashes the uncompressed tar. Collapsing them to one hash would force the transport format into the runtime’s identity, meaning a re-compression (a new gzip level, a switch to zstd) would needlessly invalidate every on-disk layer. The two-hash design lets the same filesystem layer travel under different compressions while staying one layer on disk — and it is precisely why pinning the compressor matters for cross-runner cache, even though correctness never breaks.
chain_id: position is part of identity
Why can’t overlay2 (the union filesystem Docker uses on Linux) simply share a layer wherever it sees the same bytes? Because a layer is a diff, not a standalone filesystem — its meaning depends on what sits below it. A diff_id identifies a layer’s contents, but overlay2 does not stack contents in isolation — a layer means something only on top of the specific layers beneath it. So the runtime computes a chain_id: a cumulative hash folding each diff_id together with the chain_id of everything below it. Roughly, chain_id(n) = sha256(chain_id(n-1) + " " + diff_id(n)), with chain_id(0) = diff_id(0). The consequence is sharp: the same tar layer placed at a different stack position gets a different chain_id, so overlay2 treats it as a different on-disk layer and will not share it.
This is why “just reorder the Dockerfile for clarity” can quietly cost you disk and cache. Move a COPY so the layers below it differ, and every layer above gets a new chain_id even if its own diff_id is unchanged — the contents are shared-eligible, but the stack context is not. Two images sharing a 640 MB base only dedup that base on disk when the base sits at the same position with the same lineage beneath it; break the lineage and you re-materialize layers you thought were shared.
Image A is [base, deps, app]. Image B is [base, telemetry, deps, app] — same deps and app tars, but a telemetry layer inserted under deps. Does overlay2 share the deps and app layers on disk between A and B?
- 01Distinguish a layer's diff_id from its descriptor digest: what is each hashed over, where does each appear, and which subsystem keys on it?
- 02Explain chain_id and why reordering or inserting a layer can cost on-disk sharing even when diff_ids are unchanged.
A layer is a tar of filesystem changes, and it carries two sha256 identities for two subsystems. The diff_id is the hash of the uncompressed tar; it lives in the config’s rootfs.diff_ids and is what the runtime and overlay2 verify and stack on. The descriptor digest is the hash of the compressed tar+gzip blob; it lives in the manifest’s layers array and is what the registry and the pull path content-address. One physical layer, two hashes — and the split is deliberate, because transport (compressed) and filesystem (uncompressed) want independent identities. That independence is also the trap: the registry addresses the compressed bytes, so two builders that gzip a byte-identical tar with different zlib versions or levels produce different descriptor digests, the registry stores the layer twice, and every downstream image takes a cache miss on a layer whose actual contents never changed — the 92%-to-31% collapse in the Hook, fixed by pinning the compressor in the build image. The second identity, chain_id, is a cumulative hash folding each diff_id with the chain_id of everything below it, and overlay2 keys its on-disk layers on chain_id rather than bare diff_id. So the same tar at a different stack position — reorder a Dockerfile, insert a telemetry layer under your deps — gets a new chain_id and is re-materialized on disk, costing sharing you assumed was free. Identical contents earn registry-blob dedup; identical contents plus identical lineage earn on-disk dedup. Know which digest each subsystem keys on, and the cache behaves; confuse them, and it silently doubles your bytes. Now when you see a cache-hit rate collapse after a runner migration, check whether the compressor changed before assuming the source did.
Practice
Start at the top. Tasks go easiest → hardest: recall a fact, apply it to a case, then a senior-level stretch. Open one, attempt it, then reveal.
Something unclear?
Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.