Cgroups: the resource budget the kernel actually enforces

Controllers: one tree, metered resources

Before you set a single limit, ask yourself: what happens when that limit is crossed? The answer is different for CPU and memory — and conflating them is the single most common cgroups mistake in production.

A control group (cgroup) is a node in a tree of processes that the kernel accounts and limits per controller. Modern systems run cgroup v2, a single unified hierarchy (one mount at /sys/fs/cgroup) replacing v1’s separate per-controller trees. The controllers a container runtime cares about:

• cpu — cpu.max is "QUOTA PERIOD" in microseconds: "50000 100000" means 50 ms of CPU per 100 ms window = 0.5 cores. cpu.weight (default 100) sets proportional share under contention.
• memory — memory.max is the hard limit in bytes; memory.high is a soft throttle that triggers reclaim before the wall; memory.low protects a minimum from reclaim.
• pids — pids.max caps the number of processes/threads, the direct guard against a fork bomb or a leaking entrypoint (see the previous lesson).
• io — io.max rate-limits block IO (BPS and IOPS) per device.

Together, these four controllers cover every resource a noisy neighbor can weaponize: CPU time, memory, process slots, and disk throughput. Miss any one of them and you’ve left that dimension unmetered — a container with no pids.max, for example, can fork-bomb the host as effectively as one with no memory.max.

runc writes the container’s limits from the OCI config.json linux.resources block into these files at startup, then moves the container’s PID 1 into the cgroup so every descendant is accounted together. docker run -m 512m --cpus=0.5 --pids-limit=200 becomes memory.max=536870912, cpu.max="50000 100000", pids.max=200. Reading memory.current or cpu.stat from inside or outside the cgroup is how monitoring sees real consumption — not what free reports, which (pre-cgroup-aware tooling) still shows the host.

/sys/fs/cgroup/system.slice/docker-<id>.scope/
├── cpu.max          50000 100000          # 0.5 core
├── cpu.stat         usage_usec, nr_throttled, throttled_usec
├── memory.max       536870912             # 512 MiB hard wall
├── memory.high      483183820             # ~90%, soft throttle
├── memory.current   421330944             # live charge
├── memory.events    oom 0  oom_kill 0     # increments on each OOM-kill
└── pids.max         200

CPU is a quota; memory is a wall

The two limits fail in opposite ways, and conflating them is the classic mistake. CPU is throttled, never killed: if a container exceeds its cpu.max quota within a period, the kernel simply stops scheduling it until the next period — latency spikes, nr_throttled climbs, but the process lives. Memory is a hard wall: cross memory.max and there is no “throttle harder” — the kernel runs reclaim, and if it can’t free enough, the cgroup-scoped OOM killer SIGKILLs a process within the cgroup. That is the 137 from the Hook. memory.high exists precisely to soften this: set below memory.max, it throttles allocations and forces reclaim early, turning a hard kill into back-pressure — but it is not a guarantee, only memory.max is the wall.

Why this is isolation, not virtualization

When you think of “container isolation,” you might picture something like a VM — a private pool of resources that can’t be touched from outside. That mental model will mislead you here.

A cgroup does not give a container its own CPU or own RAM — it gives it a metered slice of the host’s. There is no second pool of memory behind a hypervisor; memory.current charges the same physical pages the host kernel manages, and an unlimited container (memory.max=max) competes for all of host RAM. This is the whole point of containers being cheap: no duplicated kernel, no pre-reserved guest RAM, near-zero overhead — and the whole reason they are not VMs. The kernel is shared, so the budget, not a boundary, is what stands between a noisy neighbor and the rest of the node. Set the budgets, or a single container with no memory.max can drive the host into global OOM and take down every other container on the box.