Storage drivers: overlay2, copy-up granularity, and the inode trap
The storage driver decides how image layers stack and how the writable layer copies up. overlay2 is the default; its file-level copy-up, inode pressure, and page-cache sharing are what let you diagnose a node whose disk reads full but du reads empty.
The CI fleet started failing builds with no space left on device — but df -h showed /var/lib/docker at 38% used. The on-call engineer spent an hour chasing a phantom disk-full before someone ran df -i: inodes were at 100%. The build images bundled a node_modules tree with 180,000 tiny files, every CI container copied a chunk of it up into its writable layer, and a few hundred concurrent builds had exhausted the filesystem’s inode table while leaving 60% of the blocks free. The disk had plenty of space for bytes and none for files. The storage driver — overlay2, working at file granularity — was the mechanism, and the only durable fix was understanding what the driver does under load instead of trusting a single df -h that measured the wrong resource.
What a storage driver is, and why overlay2 won
You rarely choose a storage driver — overlay2 is picked for you. But when a disk incident lands in your lap, knowing what the driver does is the difference between a one-minute diagnosis and an hour of chasing phantoms. The storage driver is the engine subsystem that implements the layered filesystem: how read-only image layers are stacked, how the writable layer is materialized, and what copy-on-write does on write. Docker has shipped several over the years — aufs, devicemapper, btrfs, zfs, vfs — but the answer for virtually every modern Linux host is overlay2, the default. It is built on the kernel’s OverlayFS, requires no special block-device setup, works on ext4 and xfs (xfs needs ftype=1), and is the only driver actively recommended. The others are legacy: devicemapper is removed, aufs is gone from modern kernels, and vfs exists only as a no-CoW fallback that copies every layer in full (used inside containers or for debugging, never for performance).
$ docker info --format '{{ .Driver }}'
overlay2
$ docker info --format '{{ json .DriverStatus }}'
[["Backing Filesystem","extfs"],["Supports d_type","true"],["Using metacopy","false"]]Choosing a driver is mostly a non-decision today — take overlay2 — but knowing which you have matters when diagnosing, because each driver’s copy-up unit and overhead differ. vfs has no copy-up cost because it has no sharing (it duplicates everything, so a 10-layer image takes 10x the disk). btrfs and zfs copy up at block granularity and offer snapshots, but demand a dedicated filesystem and bring their own operational weight. overlay2’s tradeoff — file-level copy-up, page-cache sharing, no special setup — is what makes it the pragmatic universal default.
overlay2’s granularity: the double-edged file-level copy-up
overlay2 works at the file level, not the block level. This single fact drives both its best and worst behaviors. On the good side, it enables page-cache sharing: when ten containers from the same image read the same file out of the shared lowerdir, the kernel keeps one page-cache entry, not ten — so a node running 200 copies of an image does not pay 200x the memory for shared pages. That is why overlay2 is the recommended driver for high-density, PaaS-style hosts.
On the bad side, file-level granularity means the copy-up unit is the whole file. Modify one byte of a large file and the entire file is copied to upperdir first (covered in the writable-layer lesson) — but the subtler cost is inode and metadata pressure. Every file copied up consumes a fresh inode in the writable layer; an image with hundreds of thousands of small files (a node_modules, a Python site-packages, a sprawling asset tree) can exhaust the host filesystem’s inode table long before it exhausts blocks. df -h measures blocks and reads fine; df -i measures inodes and reads 100%. This is the failure in the Hook, and it is invisible to anyone who only watches byte-level disk usage.
A CI host reports 'no space left on device' while df -h shows /var/lib/docker at 38% used. The images bundle huge node_modules trees. What is the most likely cause?
Diagnosing and avoiding driver-level trouble
The senior moves follow from the mechanism. First, measure both resources: a node alert for disk should check df -h and df -i; an inode-exhaustion incident looks identical to a block-full incident at the application layer but has a different cure. Second, keep large mutable files and many-small-file churn out of the writable layer — volumes bypass the driver entirely, so a database on a volume pays no copy-up and burns no overlay2 inodes. Third, reclaim aggressively: stopped containers, dangling images, and build cache accumulate layers and inodes; docker system df shows the breakdown and docker system prune reclaims it, but on a busy CI host this must be scheduled, not hoped for. Fourth, on xfs backing filesystems, confirm ftype=1 (overlay2 refuses to run without it) and remember that some xfs setups have a fixed inode allocation that a file-heavy workload can outgrow.
Together these four moves cover both failure shapes: the block-waste of a large file copy-up and the invisible inode drain of a small-file workload. Skip the second and you will fix bytes while running out of files; skip the third and the problem comes back within days on a busy CI fleet.
The thing to internalize is that the storage driver is not a tuning knob to twiddle — overlay2 is the answer — but a mechanism whose behavior you must be able to reason about. Most “Docker disk” incidents are really overlay2’s file-level copy-up meeting either a large file (block waste, copy-up stalls) or a multitude of small files (inode exhaustion), and the volume is the escape hatch for both because it sidesteps the driver completely.
Why does running 200 containers from one image on an overlay2 host not cost 200x the memory for the files they share?
▸Why this works
Why did the ecosystem converge on a file-level driver when block-level CoW (btrfs, zfs) is finer-grained and avoids whole-file copy-up? Because overlay2 needs no special storage setup — it runs on a normal ext4 or xfs root with an in-kernel module — while block-level drivers demand a dedicated filesystem, snapshot management, and their own operational expertise. For the overwhelmingly common case (read-mostly image layers, state pushed to volumes) file-level copy-up is rarely on the hot path, and the page-cache sharing it enables is worth more than the block-level precision you would gain. The right answer to “which driver” is almost always “overlay2, and put your writes in a volume.”
- 01What does a storage driver do, why is overlay2 the default, and how do the alternatives differ?
- 02Explain overlay2's file-level granularity as a tradeoff: what it buys and what it costs, with the inode failure mode.
The storage driver is the engine subsystem that turns image layers into a working root filesystem: it stacks the read-only layers, materializes the writable layer, and defines what copy-on-write does. Across a decade of drivers — aufs, devicemapper, btrfs, zfs, vfs — the modern answer is overlay2, the default and the only one actively recommended. It runs on an ordinary ext4 or xfs root (xfs needs ftype=1) through the in-kernel OverlayFS with no dedicated block device, while the alternatives are legacy (devicemapper removed, aufs gone), a no-sharing debugging fallback (vfs duplicates every layer in full), or heavier block-level options (btrfs, zfs) that demand their own filesystem. The one fact that explains overlay2’s whole personality is that it copies up at file granularity, not block granularity. That granularity buys page-cache sharing: many containers reading the same file from the shared lowerdir share a single page-cache entry, so a node can run hundreds of copies of an image without multiplying memory — the reason overlay2 suits high-density hosts. The same granularity is the cost: modifying one byte of a large file copies the entire file up, and every copied-up file burns a fresh inode, so a many-small-file image can exhaust the inode table while blocks sit mostly free. That is the classic phantom disk-full — df -h reads fine, df -i reads 100% — and the only way to see it is to measure both. The senior posture is not to tune the driver but to reason about it: overlay2 is the answer, large mutable files and file-heavy churn belong on volumes that bypass it entirely, and disk incidents on Docker hosts are usually file-level copy-up meeting either one big file or a million small ones. Now when you see “no space left on device” on a Docker host, check df -i before you check df -h — inodes are the invisible resource overlay2 quietly drains.
Practice
Start at the top. Tasks go easiest → hardest: recall a fact, apply it to a case, then a senior-level stretch. Open one, attempt it, then reveal.
Something unclear?
Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.