Pull-through caches and mirrors: surviving rate limits without serving stale images

What a pull-through cache actually does

If your fleet pulls images from Docker Hub directly, you’re paying the bandwidth cost on every pull and sharing one rate-limit quota across your entire cluster. A pull-through cache solves both problems with a single mechanism — and understanding exactly how it works tells you where it can fail. A pull-through cache (registry mirror) is an ordinary registry running in proxy mode: configured with an upstream (proxy.remoteurl: https://registry-1.docker.io), it answers the same /v2/ pull API, but on a request it checks local storage first and only reaches upstream on a miss, storing what it fetches. The leverage is entirely the content-addressed blob layer: because a layer blob’s name is its sha256: digest, once any node has pulled node:20 through the mirror, the mirror holds those layer blobs, and every subsequent node’s GET /v2/.../blobs/sha256:… is a local hit served at LAN bandwidth. A 300-node rollout of a 180 MB image stops being 300 × 180 MB ≈ 54 GB of upstream transfer and becomes one ~180 MB upstream fetch plus 299 LAN reads. The clients point at it by setting registry-mirrors in the Docker daemon config (or containerd’s registry host config); to the client it is just a faster registry.

// /etc/docker/daemon.json — clients fetch via the in-cluster mirror first
{ "registry-mirrors": ["https://mirror.internal:5000"] }

The two payoffs are bandwidth and rate limits, and the rate-limit one is what bites first. Docker Hub enforces 100 pulls per 6 hours for anonymous and 200 per 6 hours for authenticated free accounts, counted per source IP (or per account when logged in) and per manifest pull. A NATed cluster shares one egress IP, so its whole fleet shares one quota — the Hook’s failure. A pull-through cache fronts upstream with one identity: the fleet’s thousands of pulls become a handful of upstream manifest fetches, keeping you under the limit by construction.

The failure mode: caching the mutable half

Here is the trap that turns a mirror from a savior into an incident. A pull resolves a tag to a manifest first, then fetches blobs. The blobs are safe to cache forever. But if the mirror also caches the tag→manifest resolution and serves it without revalidating, then after upstream re-pushes node:20 to a new digest, the mirror keeps handing out the old manifest — and since it still has the old manifest’s blobs locally, every node happily assembles a stale image with zero errors. You get the inverse of the lesson-2 incident: there, a tag moved and everyone got the new image; here, a tag moved and your mirror pins everyone to the old one. The correct behavior is to treat tag resolution as always revalidate against upstream (a conditional manifest request; blobs, being immutable, never need revalidation) — or, far more robustly, have clients pull by digest, which makes the cache trivially correct because a digest request can only ever be a hit for the exact right bytes or a miss to fetch them.

Mirrors, sizing, and the operational picture

A few senior points round this out. A mirror can mean two related things: a pull-through cache (lazy, fills on demand) or a fully replicated registry (eager copy, for geo-distribution or air-gapped sites). For rate-limit and bandwidth relief inside one cluster, the lazy pull-through cache is the right tool and the cheaper one. Sizing matters: the cache needs disk for the working set of layers (a few hundred GB covers most fleets, since base images dominate and dedup by digest is automatic) and you must run garbage collection on it like any registry, or it grows unbounded. Cache hit ratio is the metric that proves the win — a healthy fleet mirror sits well above 90% hits on blob requests once base images are warm, and that ratio is exactly the upstream-traffic and rate-limit reduction. And the authenticated-upstream nuance: pointing the mirror at Docker Hub with credentials raises your shared limit to the authenticated tier and consolidates the whole fleet under one accountable identity, which is both higher and observable. The throughline: cache the immutable blob layer aggressively, keep the mutable tag layer honest, and a single shared cache turns a fleet-scale pull storm into one upstream fetch.