cgroups and resource limits

rlimits: per-process resource ceilings enforced by the kernel.

Every process inherits a set of rlimits from its parent. Each limit has a soft value (the current enforced ceiling, which the process may lower but not raise above hard) and a hard value (the ceiling for the soft limit — only root can raise it).

# View current shell's rlimits:
ulimit -a
# core file size          (blocks, -c) 0
# data seg size           (kbytes, -d) unlimited
# scheduling priority             (-e) 0
# file size               (blocks, -f) unlimited
# pending signals                 (-i) 63296
# max locked memory       (kbytes, -l) 8192
# max memory size         (kbytes, -m) unlimited
# open files                      (-n) 1024    ← RLIMIT_NOFILE soft
# pipe size            (512 bytes, -p) 8
# POSIX message queues     (bytes, -q) 819200
# real-time priority              (-r) 0
# stack size              (kbytes, -s) 8192
# cpu time               (seconds, -t) unlimited
# max user processes              (-u) 63296   ← RLIMIT_NPROC
# virtual memory          (kbytes, -v) unlimited
# file locks                      (-x) unlimited

# The two limits you will hit most often in production:

# RLIMIT_NOFILE: maximum number of open file descriptors
# Default soft: 1024. Default hard: 1048576 (on modern Ubuntu).
# Hit it and you get: "Too many open files" (errno EMFILE)
# A Node.js or Java server under load needs far more than 1024 FDs.

# RLIMIT_NPROC: maximum number of processes/threads the user may have
# Hit it and fork() returns EAGAIN: "Resource temporarily unavailable"
# Fork bombs are stopped here.

Setting rlimits: ulimit for the session, /etc/security/limits.conf for persistence.

# Raise the open file limit for the current shell and its children:
ulimit -n 65536
# Verify:
ulimit -n
# 65536

# This only affects the current shell session.
# For a service, set it in /etc/security/limits.conf:
# (format: <domain> <type> <item> <value>)

cat /etc/security/limits.conf
# *         soft  nofile  65536
# *         hard  nofile  1048576
# www-data  soft  nproc   4096
# www-data  hard  nproc   8192
#
# * = all users; soft = soft limit; hard = hard limit

# For systemd-managed services, prefer LimitNOFILE in the unit file:
# [Service]
# LimitNOFILE=65536
# This is the right way — limits.conf is bypassed for services that do
# not go through PAM (i.e., most systemd services).

# Check effective limits of a running process (PID 1234):
cat /proc/1234/limits
# Limit                     Soft Limit   Hard Limit   Units
# Max open files            65536        1048576      files
# Max processes             63296        63296        processes
# Max locked memory         8388608      8388608      bytes

cgroups v2: the unified hierarchy.

cgroups v2 (the current default on Ubuntu 22.04+) uses a single unified filesystem hierarchy at /sys/fs/cgroup. Every process belongs to exactly one cgroup. Child cgroups inherit from parents. systemd maps services to cgroups automatically.

# See your cgroup hierarchy:
systemd-cgls
# Control group /:
# -.slice
#   ├─system.slice
#   │ ├─nginx.service
#   │ │ ├─987 nginx: master process
#   │ │ └─988 nginx: worker process
#   │ └─ssh.service
#   │   └─1234 sshd: root@pts/0
#   └─user.slice
#     └─user-1000.slice
#       └─session-1.scope
#         └─2345 -bash

# See the cgroup of any process:
cat /proc/$$/cgroup
# 0::/user.slice/user-1000.slice/session-1.scope

# Inspect resource controllers available on the system:
cat /sys/fs/cgroup/cgroup.controllers
# cpuset cpu io memory hugetlb pids rdma misc
# 'cpu' and 'memory' are the two you will use most

# Current memory usage of a cgroup:
cat /sys/fs/cgroup/system.slice/nginx.service/memory.current
# 45678592   (bytes — about 44 MB)

# Current CPU usage in microseconds:
cat /sys/fs/cgroup/system.slice/nginx.service/cpu.stat
# usage_usec 123456789
# user_usec  100000000
# system_usec 23456789

systemd CPUQuota and MemoryMax: setting limits on a service.

systemd translates unit-file directives into cgroup writes. You almost never need to write to /sys/fs/cgroup directly.

# Set a memory cap and CPU quota on an existing service:
sudo systemctl edit nginx
# This opens a drop-in editor. Add:
# [Service]
# MemoryMax=512M
# CPUQuota=50%

# MemoryMax: cgroup memory.max — hard cap. When the cgroup exceeds this,
# the kernel invokes the cgroup-local OOM killer (kills a process in this
# cgroup). NOT the system-wide OOM killer.

# CPUQuota=50%: the cgroup gets at most 50% of one CPU core per period
# (default period is 100ms). On a 4-core machine, 50% means one core at
# full rate, NOT 50% of all 4 cores.

# Reload:
sudo systemctl daemon-reload
sudo systemctl restart nginx

# Verify the limits are applied:
systemctl show nginx | grep -E 'MemoryMax|CPUQuota'
# MemoryMax=536870912
# CPUQuotaPerSecUSec=500000   (500ms out of 1000ms = 50%)

# Or read directly from the cgroup:
cat /sys/fs/cgroup/system.slice/nginx.service/memory.max
# 536870912
cat /sys/fs/cgroup/system.slice/nginx.service/cpu.max
# 50000 100000   (50ms budget per 100ms period = 50%)

# Transient limits (no unit file change, useful for testing):
sudo systemctl set-property nginx.service MemoryMax=256M