The registry pull/push protocol: manifests, blobs, and content-addressed distribution
A registry is a content-addressed blob store behind a small HTTP API. Pull resolves a tag to a manifest, fetches a config and layer blobs by sha256 digest; push uploads blobs (chunked or monolithic, or cross-repo mounted) then PUTs the manifest. Digests are the real identity.
The on-call page said “image pull backoff” across the whole fleet, and the registry dashboard was green. What was actually happening showed up only in a packet capture: every node was issuing GET /v2/payments-api/manifests/v2.4.1, getting a 200 with a tiny JSON body, then a storm of GET /v2/payments-api/blobs/sha256:... requests — and one of those blob GETs was returning 404. A garbage-collection job had run against the registry the night before, and because of a race between an in-flight push and the mark phase, it had deleted a layer blob that a still-referenced manifest pointed at. The manifest resolved fine; the tag looked healthy; the registry’s own health check fetched no blobs. But the image was a directory listing pointing at a file that no longer existed. The fix was understanding that a registry is not “where images live” — it is a content-addressed blob store with a thin index on top, and the index had outlived its contents.
Two object kinds, one address space
Before you can debug a pull failure or reason about garbage collection, you need a mental model of what a registry actually stores — because it is not what the word “registry” suggests. Everything in a registry is one of two things: a manifest (small JSON describing an image — its config plus an ordered list of layers) or a blob (an opaque content-addressed object: a layer tarball or the config JSON). Both live under the /v2/ API, the OCI distribution spec’s single namespaced tree:
GET /v2/<name>/manifests/<reference> # reference = a tag (v2.4.1) OR a digest (sha256:ab…)
GET /v2/<name>/blobs/<digest> # digest is ALWAYS sha256:<hex>, never a tag
HEAD /v2/<name>/blobs/<digest> # does the layer already exist here? (used before push)The split is the whole design. A blob’s address is its content: sha256:9b2a… is the SHA-256 of the bytes, so a blob is immutable by definition — change one byte and it is a different object with a different name. A manifest’s reference can be a tag, and a tag is just a mutable pointer the registry stores in its index. So myrepo:latest and myrepo@sha256:9b2a… are categorically different: the digest names an exact bit-identical image forever; the tag names “whatever was last pushed under this label.” When you pull a tag, the client does a GET .../manifests/v2.4.1, reads the returned Content-Type to learn the manifest’s media type (e.g. application/vnd.oci.image.manifest.v1+json for a single-arch image, or ...image.index.v1+json for a multi-arch index that lists per-platform manifests), then fetches the config blob and each layers[].digest blob — deduplicating any layer it already has in its local content store.
▸Why this works
Why address blobs by hash instead of a path or an id? Because content-addressing makes deduplication and integrity free. Two images sharing a base layer reference the identical sha256: blob — the registry stores it once, the client downloads it once, and a CDN caches it forever because the URL can never mean different bytes. Integrity comes for the same reason: the client recomputes the SHA-256 of every blob it pulls and compares it to the digest it asked for, so a corrupted or tampered byte is caught locally, no signature needed for the transport-integrity layer. The cost is that nothing about a tag is verifiable — only a digest is.
Push: blobs first, manifest last
When you push an image, what order does your client upload things? If you get it wrong the registry rejects the manifest — and knowing the correct order also explains why cross-repo mounts and chunked uploads work the way they do. Push is the mirror image of pull and the order is load-bearing. You cannot PUT a manifest that references a blob the registry does not yet have, so the client uploads blobs first, manifest last:
POST /v2/<name>/blobs/uploads/ # start an upload, get a session URL (Location:)
PATCH <session-url> # chunked upload: send a byte range, repeat
PUT <session-url>?digest=sha256:<hex> # finalize: registry verifies the bytes hash to <digest>
PUT /v2/<name>/manifests/<tag> # only after every referenced blob is presentTwo senior optimizations live here. Cross-repository blob mount: before uploading a layer the client HEADs it; if the registry already holds that sha256: (in any repo the credential can read), the client issues POST .../blobs/uploads/?mount=<digest>&from=<other-repo> and the layer is linked without transferring a byte — the reason pushing a new tag of a base-image-sharing app moves only the changed layers, often a few MB instead of hundreds. Chunked vs monolithic upload: small blobs go in one PUT with the body inline; large blobs stream as PATCH ranges so a dropped connection resumes from the last acknowledged offset instead of restarting a 400 MB layer. The finalizing PUT ...?digest= is where the registry verifies — it recomputes the hash and rejects the upload with 400 DIGEST_INVALID if the bytes do not match, so a corrupted transfer can never be committed under a valid name.
During a pull, GET /v2/app/manifests/v2.4.1 returns 200 with a valid manifest, but a subsequent GET /v2/app/blobs/sha256:9b2a… returns 404. What does this state mean?
Why the protocol shapes operations
Three operational facts fall straight out of this design — and when you hit any of these problems in production, recognizing which design choice caused them tells you where the fix lives. First, pull is fan-out: one manifest GET spawns N+1 blob GETs (N layers plus the config), so a 12-layer image is ~13 requests per node, and a 200-node rollout is thousands of blob GETs — which is exactly why pull-through caches and rate limits exist (a public registry like Docker Hub caps anonymous pulls at 100 per 6 hours and authenticated free pulls at 200 per 6 hours, counted per manifest pull, throttling fleet rollouts that re-pull). Second, garbage collection is a two-phase mark-and-sweep over the blob store: it walks every manifest to mark referenced digests, then deletes unmarked blobs — and a push racing the mark phase can leave a referenced blob unmarked and swept, the exact Hook failure. Run GC with the registry in read-only mode for correctness. Third, digests are the only safe deploy reference: pinning a Deployment to app@sha256:… means the bytes can never change under you, whereas app:latest can be re-pushed to point at a different manifest between two nodes pulling it, so a single rollout runs two different images.
A client is about to push a new app tag whose layers are mostly shared with a base image already in the registry. Why does the push transfer only a few megabytes instead of the full image?
- 01Walk through what a docker pull of app:v2.4.1 actually does at the HTTP level, and where verification happens.
- 02Explain why a tag and a digest are not interchangeable as deploy references, using the registry's data model.
A registry is not a filesystem of images; it is a content-addressed blob store with a thin mutable index, exposed through the OCI distribution spec’s /v2/ API. Two object kinds exist: manifests (small JSON listing a config blob and ordered layer blobs, addressable by tag or by digest) and blobs (opaque content-addressed objects whose sha256 name is the hash of their bytes, so they are immutable by construction). A pull is GET /v2/<name>/manifests/<ref> — reading the returned media type to tell a single image from a multi-arch index — followed by GET /v2/<name>/blobs/<digest> for the config and every not-yet-held layer, each verified locally against its digest. A push runs the mirror order, blobs first: POST to open an upload session, PATCH chunked ranges for resumable large-layer transfer, PUT ?digest= to finalize where the registry verifies the hash, and HEAD plus ?mount= to skip layers the registry already holds via cross-repository blob mount, transferring only the changed bytes. From this model fall the operational truths a senior carries: pull is fan-out (N+1 blob GETs per image, why pull-through caches and the 100/200-per-6h Docker Hub rate limits exist), garbage collection is a mark-and-sweep that must run read-only or it can sweep a still-referenced blob and strand a manifest, and only a digest — never a tag — is a safe deploy reference, because a tag is a pointer that can be re-aimed under a running rollout. Now when you see “image pull backoff” or a 404 on a blob while the manifest returns 200, you know exactly which layer of this model broke — and where to look first.
Practice
Start at the top. Tasks go easiest → hardest: recall a fact, apply it to a case, then a senior-level stretch. Open one, attempt it, then reveal.
Something unclear?
Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.