Filesystems
A filesystem maps filenames and directories onto disk blocks. ext4 is the safe default; xfs scales for large files; btrfs adds copy-on-write and snapshots. Inodes store metadata — and you can run out of inodes while the disk still has free space.
You have a blank partition and you want to store files on it. The partition is just a sequence of raw bytes — it has no concept of “file” or “directory”. A filesystem is the layer that imposes that structure: it decides how to carve those bytes into files, where to store filenames, how to track which blocks belong to which file, and how to recover when a write is interrupted by a power cut.
The choice of filesystem matters in production: the wrong one for your workload causes performance cliffs, and one obscure failure mode — running out of inodes while df still shows gigabytes free — will confuse you deeply if you haven’t seen it before.
After this lesson you can explain what a filesystem does and what an inode is, create a filesystem with mkfs, describe the key differences between ext4, xfs, and btrfs, and diagnose the “no space left on device” error that occurs when inodes are exhausted despite free disk space.
A filesystem maps filenames and directories onto disk blocks. When you write a file, the filesystem decides which blocks to use, writes your data there, and records the mapping in its metadata structures. When you read the file back, it looks up the mapping and reassembles the data.
The core structures:
- Superblock — the master record: filesystem type, total size, block size, number of inodes, pointers to the block group descriptors. If the superblock is corrupted, the filesystem cannot be mounted. ext4 keeps backup superblock copies in several block groups for exactly this reason.
- Block groups — ext4 divides the partition into groups of blocks. Each group has its own copy of the block bitmap (which blocks are free), inode bitmap, and inode table. This locality means that reading a file usually hits one block group, not random disk locations.
- Inode (index node) — a fixed-size metadata record (256 bytes in ext4) that stores everything about a file except its name: owner UID/GID, permissions, timestamps (atime/mtime/ctime), size, and pointers to the data blocks.
- Directory entry (dentry) — the mapping from a filename to an inode number. A directory is itself a file whose data is a list of
(filename, inode_number)pairs.
# See filesystem metadata on a mounted partition
stat /etc/hostname
# File: /etc/hostname
# Size: 10 Blocks: 8 IO Block: 4096 regular file
# Device: 8,3 Inode: 2621554 Links: 1
# Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
# Access: 2024-06-20 09:12:01
# Modify: 2024-01-15 14:22:03
# Change: 2024-01-15 14:22:03
# The inode number is 2621554 — that is the filesystem's internal ID for this file
# The filename '/etc/hostname' is just a dentry pointing to this inodeext4 is the default Linux filesystem and the safe choice for most workloads. It has been the default on Debian/Ubuntu since 2010, has excellent fsck tooling, and handles power failures gracefully via journaling.
# Create an ext4 filesystem on a partition
sudo mkfs.ext4 /dev/sda3
# mke2fs 1.47.0 (5-Feb-2023)
# Creating filesystem with 130678784 4k blocks and 32669696 inodes
# Filesystem UUID: e5f6a7b8-c9d0-1234-5678-abcdef012345
# Superblock backups stored on blocks: 32768, 98304, 163840, ...
# Allocating group tables: done
# Writing inode tables: done
# Creating journal (131072 blocks): done
# Writing superblocks and filesystem accounting information: done
# Custom inode count (useful for directories-heavy workloads like mail servers)
sudo mkfs.ext4 -N 8000000 /dev/sda3 # 8 million inodes instead of default
# Custom block size (4096 is the default; match your storage's native sector size)
sudo mkfs.ext4 -b 4096 /dev/sda3
# Add a label (useful for fstab and identification)
sudo mkfs.ext4 -L data /dev/sda3
# Inspect filesystem metadata after creation
sudo tune2fs -l /dev/sda3 | grep -E 'Inode count|Block count|Block size|Journal'
# Inode count: 32669696
# Block count: 130678784
# Block size: 4096
# Journal inode: 8Journaling is what makes ext4 reliable. Before writing to the filesystem data structures, it writes a record to the journal (a dedicated area on disk). If the system crashes mid-write, fsck replays the journal to restore consistency instead of scanning the entire disk.
xfs and btrfs cover workloads where ext4 is not the best fit.
# Create xfs (large-file, high-throughput workloads: databases, video, streaming)
sudo mkfs.xfs /dev/sdb1
# meta-data=/dev/sdb1 isize=512 agcount=4, agsize=65536 blks
# data = bsize=4096 blocks=262144, imaxpct=25
# naming =version 2 bsize=4096 ascii-ci=0, ftype=1
# log =internal log bsize=4096 blocks=16384, version=2
# realtime =none
# Create btrfs (snapshots, subvolumes, transparent compression, checksums)
sudo mkfs.btrfs /dev/sdc1
# btrfs-progs v6.1
# Label: (none)
# UUID: 9f8e7d6c-...
# Nodesize: 16384
# Sectorsize: 4096
# Features: skinny-metadata, no-holesKey differences:
| Feature | ext4 | xfs | btrfs |
|---|---|---|---|
| Journaling | yes (metadata + optional data) | yes (metadata) | copy-on-write (no journal needed) |
| Max file size | 16 TiB | 8 EiB | 16 EiB |
| Max filesystem size | 1 EiB | 8 EiB | 16 EiB |
| Snapshots | no (needs LVM) | no (needs LVM) | yes, built-in, instant |
| Shrink online | no | no | no (offline only) |
| Grow online | yes | yes | yes |
| Best for | general purpose, boot, home | large files, databases, high I/O | snapshots, dedupe, NAS, home servers |
| Ubuntu default | yes (root) | yes (RHEL default) | Fedora default since F33 |
btrfs write performance can lag ext4/xfs on spinning disks under write-heavy loads due to copy-on-write overhead. On SSDs the difference is small.
Inodes are a fixed pool. You can run out of them while the disk still has space. This is one of the most confusing errors an operator encounters.
# Check inode usage — THIS IS THE CHECK PEOPLE FORGET
df -i
# Filesystem Inodes IUsed IFree IUse% Mounted on
# /dev/sda3 32669696 3100000 29569696 9% /
# /dev/sda2 131072 78000 53072 60% /boot <- getting full
# /var/spool/mail 524288 524287 1 100% /var/mail <- FULL!
# What happens when inodes are exhausted:
# touch /var/mail/newfile
# touch: cannot touch '/var/mail/newfile': No space left on device
# BUT:
df -h /var/mail
# Filesystem Size Used Avail Use% Mounted on
# /dev/sdc1 20G 8.1G 11.9G 41% /var/mail <- 41% free! But no inodes left.
# Count files in a suspicious directory
find /var/spool/postfix/deferred -type f | wc -l
# 524281 <- mail server queued 524281 emails — each consumed one inode
# How many inodes were allocated at mkfs time?
sudo tune2fs -l /dev/sdc1 | grep 'Inode count'
# Inode count: 524288 <- mkfs.ext4 calculated based on total size, not workloadThe fix is to either clean up the files consuming inodes, or reformat with a higher inode count (mkfs.ext4 -N <count>). You cannot add inodes to an existing ext4 filesystem without reformatting. xfs and btrfs allocate inodes dynamically and do not have this problem — they only run out of inodes when they run out of disk space.
The superblock is the filesystem’s master record — and knowing where backups are saves you.
# Find backup superblock locations (useful when primary is corrupted)
sudo mke2fs -n /dev/sda3 # -n = dry run, shows what mkfs would do
# Superblock backups stored on blocks: 32768, 98304, 163840, 229376, ...
# If the primary superblock (block 0) is damaged:
sudo fsck.ext4 -b 32768 /dev/sda3 # use the backup at block 32768
# e2fsck 1.47.0 (using backup superblock at block 32768)
# /dev/sda3: recovering journal
# /dev/sda3: clean, 3100000/32669696 files, 24500000/130678784 blocks
# Check and repair a filesystem (must be unmounted first)
sudo umount /dev/sda3
sudo fsck.ext4 -f /dev/sda3 # -f = force check even if marked clean
# For xfs (xfs uses a different approach — log replay rather than fsck scan)
sudo xfs_repair /dev/sdb1 # must be unmounted
# For btrfs
sudo btrfs check /dev/sdc1 # read-only check; use --repair with extreme cautionNever run fsck on a mounted filesystem. On ext4, the kernel marks the filesystem “clean” at unmount and “dirty” at mount. At boot, systemd runs fsck on partitions marked as dirty (pass 1 or 2 in /etc/fstab). The superblock’s Last mount time and Last checked fields tell you when the filesystem was last verified.
Diagnosing “no space left on device” on a mail server that still shows free disk space.
# Alert: mail delivery failing with "no space left on device"
df -h /var/spool
# Filesystem Size Used Avail Use% Mounted on
# /dev/sdb1 50G 22G 28G 44% /var/spool <- 44% free, looks fine
# Check inodes — this is the missing step
df -i /var/spool
# Filesystem Inodes IUsed IFree IUse% Mounted on
# /dev/sdb1 3276800 3276800 0 100% /var/spool <- 100% inode usage!
# Find the directory with the most files
find /var/spool -xdev -type d | while read dir; do
echo "$(find "$dir" -maxdepth 1 -type f | wc -l) $dir"
done | sort -rn | head -5
# 3241000 /var/spool/postfix/deferred
# 35000 /var/spool/postfix/active
# 700 /var/spool/cron
# The mail queue has 3.2 million deferred messages
# Each is a separate file = one inode per message
# Immediate fix: flush or purge the deferred queue
sudo postsuper -d ALL deferred # Postfix: discard all deferred mail
# OR investigate why they are deferred:
sudo mailq | tail -5
# Long-term fix: reformat /dev/sdb1 with more inodes
# mkfs.ext4 -N 10000000 /dev/sdb1 <- 10M inodes for a mail server
# Or use xfs (dynamic inode allocation, no fixed pool)The lesson: always check df -i alongside df -h when you get “no space left on device”. On a mail server, small-file workload, or container image layer store, inode exhaustion is a realistic failure.
▸Why this works
Why did filesystem designers use a fixed inode pool instead of allocating inodes on demand? In ext2 (the predecessor to ext4, designed in 1993), inodes had to be pre-allocated to enable constant-time inode lookup by number. A dynamic allocation scheme would have required more complex on-disk structures. xfs (also 1993, from SGI) solved this differently from the start: its inode allocation is B-tree-based and fully dynamic. The ext4 journal format was designed to be backward-compatible with ext2, so the fixed-pool tradeoff was carried forward. Modern btrfs (2009) also uses dynamic allocation. If you regularly provision ext4 on workloads that create millions of small files — mail spools, build caches, container layers — specify a higher inode ratio at mkfs time: mkfs.ext4 -T news /dev/sdb1 (the news profile sets one inode per 4 KiB instead of the default one per 16 KiB).
▸Common mistake
Do not run mkfs on a device that is already mounted, even if you think the data is unimportant. The kernel has cached inode and block data for that device. Writing a new filesystem header while the old one is still mounted creates a split-brain state — the VFS layer still thinks it is reading the old filesystem while the disk now has a new superblock. This can corrupt data on adjacent partitions through confused write paths. Always unmount (umount /dev/sdb1) and verify with mount | grep sdb1 (empty output = unmounted) before running mkfs.
A server running a high-volume mail system reports 'No space left on device' when trying to create new files. df -h shows 45% disk usage. What is the most likely cause and how do you confirm it?
A filesystem imposes file/directory structure on a raw block partition. The key structures are the superblock (master metadata record), inodes (per-file metadata: owner, permissions, timestamps, block pointers), and directory entries (filename-to-inode mappings). ext4 is the Debian/Ubuntu default — journaled, stable, excellent tooling. xfs excels at large files and high throughput (RHEL default). btrfs adds copy-on-write, snapshots, and dynamic inode allocation. Create a filesystem with mkfs.ext4, mkfs.xfs, or mkfs.btrfs. A critical gotcha: ext4 pre-allocates a fixed inode pool at format time — you can exhaust it before running out of disk space. df -i shows inode usage; always check it alongside df -h when diagnosing “no space left on device”.
Practice
Start at the top. Tasks go easiest → hardest: recall a fact, apply it to a case, then a senior-level stretch. Open one, attempt it, then reveal.
Something unclear?
Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.