Persistent journals and log rotation

Volatile vs persistent: where does the journal live?

journald stores entries in one of two places depending on configuration:

# Check where the journal currently lives:
ls /run/log/journal/   # volatile — lost on reboot (tmpfs)
ls /var/log/journal/   # persistent — survives reboot

# Check the active storage mode:
journalctl --disk-usage
# Archived Journals: 0 B
# Active Journals: 48.0 M

# On a system with volatile storage only:
# /var/log/journal/ does NOT exist
# All data lives in /run/log/journal/ (tmpfs, lost on reboot)

# Verify the current Storage setting:
grep -i storage /etc/systemd/journald.conf 2>/dev/null || \
  grep -i storage /etc/systemd/journald.conf.d/*.conf 2>/dev/null || \
  echo "Storage= not set — default is auto"

The default Storage=auto means: use persistent if /var/log/journal/ already exists, otherwise use volatile. On a fresh Ubuntu install /var/log/journal/ typically does not exist, so you are volatile by default.

Enable persistent storage.

One line in journald.conf and a directory creation:

# Edit journald configuration:
sudo mkdir -p /etc/systemd/journald.conf.d/
sudo tee /etc/systemd/journald.conf.d/persistence.conf <<'EOF'
[Journal]
Storage=persistent
EOF

# Create the persistent storage directory (journald will use it on restart):
sudo mkdir -p /var/log/journal

# Apply the change (no reboot needed):
sudo systemctl restart systemd-journald

# Verify:
journalctl --disk-usage
# Archived Journals: 0 B
# Active Journals: 52.0 M  ← now on-disk in /var/log/journal/

ls /var/log/journal/
# <machine-id>/  ← one directory per machine ID

After enabling persistence, logs from the next boot and all subsequent boots survive reboots. Logs from before the change are not retroactively saved.

Capping journal size with SystemMaxUse.

Without a size cap, a persistent journal will grow until it fills the disk. journald has built-in defaults (10% of the filesystem or 4 GB, whichever is smaller), but on a small VPS that default can still eat critical space.

# Set explicit size caps in the config drop-in:
sudo tee /etc/systemd/journald.conf.d/size.conf <<'EOF'
[Journal]
SystemMaxUse=500M
SystemKeepFree=200M
RuntimeMaxUse=100M
EOF

# SystemMaxUse      — max total disk the journal may use
# SystemKeepFree    — always keep this much free on the partition
# RuntimeMaxUse     — cap for volatile (/run) journal

sudo systemctl restart systemd-journald

# Check current disk usage after restart:
journalctl --disk-usage
# Archived Journals: 320.1 M
# Active Journals: 48.0 M
# Total: 368.1 M  ← within the 500M cap

journald enforces the cap by rotating and deleting the oldest archived journal files first. You do not need to run any cleanup command manually — enforcement is continuous.

Manual vacuum: trim old journal data on demand.

When you need to reclaim space immediately (e.g. after a logging storm), use --vacuum-size or --vacuum-time:

# Remove archived journals until total usage is under 200M:
sudo journalctl --vacuum-size=200M
# Vacuuming done, freed 148.3M of archived journals

# Remove journals older than 2 weeks:
sudo journalctl --vacuum-time=2weeks

# Remove journals older than 30 days:
sudo journalctl --vacuum-time=30d

# Combine both (applies whichever is more restrictive):
sudo journalctl --vacuum-size=500M --vacuum-time=30d

# Verify space after vacuum:
journalctl --disk-usage

Vacuum only removes archived (rotated) journals, never the active one. It is safe to run while the system is live. A common pattern: run --vacuum-time=30d weekly from a systemd timer or cron job.

logrotate for plain-text logs in /var/log.

Not everything goes through journald. Apps like nginx, PostgreSQL, and many legacy daemons still write to files under /var/log. logrotate handles these.

# logrotate is driven by /etc/logrotate.conf and /etc/logrotate.d/:
cat /etc/logrotate.d/nginx
# /var/log/nginx/*.log {
#   daily
#   missingok
#   rotate 14
#   compress
#   delaycompress
#   notifempty
#   create 0640 www-data adm
#   sharedscripts
#   postrotate
#     /bin/kill -USR1 `cat /run/nginx.pid 2>/dev/null` 2>/dev/null || true
#   endscript
# }

# Key directives:
# daily/weekly/monthly   — rotation frequency
# rotate 14              — keep 14 rotated files, then delete
# compress               — gzip rotated files (saves ~90% space)
# delaycompress          — do not compress the most recent rotated file
#                           (nginx may still be writing to it)
# postrotate/endscript   — run this shell block after rotation
#                           (sends USR1 to nginx to reopen log files)

# Test logrotate without actually rotating (dry run):
sudo logrotate --debug /etc/logrotate.d/nginx

# Force rotation immediately (ignores time check):
sudo logrotate --force /etc/logrotate.d/nginx

The postrotate block is critical: apps that hold a file descriptor open will keep writing to the old (now renamed) file even after logrotate moves it. Sending USR1 (or reloading the service) tells the app to close and reopen its log file handle, so it starts writing to the new empty log.