journald and journalctl

journald captures everything systemd units write to stdout/stderr.

When you run a service under systemd, the unit’s stdout and stderr are piped directly into journald. You never need to configure a log file location — the journal is automatic.

# Show the last 50 lines of the entire journal (newest at bottom):
journalctl -n 50

# Show logs for a specific unit (the most important flag):
journalctl -u nginx.service

# Show logs for a unit from the current boot only:
journalctl -u nginx.service -b

# Combine: last 100 lines from nginx, current boot:
journalctl -u nginx.service -b -n 100

The -u flag is the workhorse. Without it you get the full system journal — hundreds of lines per second on a busy box. With it you get exactly one service.

Filter by priority to cut through noise.

journald maps syslog severity levels to an integer 0–7. -p filters to that level and above (lower number = more severe).

# Priority levels (number = syslog severity):
# 0 emerg   1 alert   2 crit   3 err
# 4 warning  5 notice  6 info  7 debug

# Show only errors and worse for nginx since last boot:
journalctl -u nginx.service -b -p err

# Show warnings and worse, all units:
journalctl -b -p warning

# Show everything including debug (very verbose):
journalctl -u myapp.service -p debug

# Filter a range (e.g. warning through err):
journalctl -u myapp.service -p warning..err

In practice -p err is the first thing you run when a service misbehaves. It drops info and notice noise and shows only what went wrong.

Time filters: --since and --until.

Boot boundaries (-b) are convenient, but sometimes you need a specific window — say, the 5 minutes before an alert fired.

# Since a specific time:
journalctl -u postgres.service --since "2024-03-15 14:00:00"

# Time range:
journalctl -u postgres.service \
  --since "2024-03-15 14:00:00" \
  --until "2024-03-15 14:10:00"

# Relative time (last 30 minutes):
journalctl -u postgres.service --since "30 min ago"

# Boot offsets: -b 0 = current boot, -b -1 = previous boot, -b -2 = two boots ago:
journalctl -u postgres.service -b -1

# List available boots:
journalctl --list-boots

--list-boots is underused. It shows every boot record with its index, so after a crash-reboot you can reach back to -b -1 and read the dying moments of the previous session.

Follow live output with -f.

-f tails the journal in real time — like tail -f but for structured logs.

# Follow all new journal entries live:
journalctl -f

# Follow only a specific unit live (the one you actually want):
journalctl -u nginx.service -f

# Follow + priority filter (only errors live):
journalctl -u myapp.service -f -p err

# Combined: follow errors from two units simultaneously:
journalctl -u nginx.service -u php-fpm.service -f -p err

You can combine -f with all the other flags. -f -p err during a deployment gives you a live stream of only fatal events without noise.

Structured fields: what makes journald different from plain-text logs.

Every journal entry is a key-value record, not a line of text. The binary format stores typed fields alongside the message. You can query on any field.

# Show verbose output with all fields for a unit:
journalctl -u nginx.service -o verbose | head -40
# Output includes:
#   _SYSTEMD_UNIT=nginx.service
#   PRIORITY=3
#   _PID=1234
#   _UID=0
#   _COMM=nginx
#   MESSAGE=connect() failed (111: Connection refused)...

# Query by a specific field value (e.g., only from PID 1234):
journalctl _PID=1234

# All entries from the www-data user:
journalctl _UID=33

# Multiple field filters (AND):
journalctl _SYSTEMD_UNIT=nginx.service _PID=1234

# Output formats: short (default), json, cat (message only):
journalctl -u nginx.service -b -p err -o json | head -5
journalctl -u nginx.service -b -o cat

Key structured fields: _SYSTEMD_UNIT (which unit), PRIORITY (0–7), _PID (process ID), _UID (user ID), _COMM (command name), MESSAGE (the actual log line). Filtering on _PID is something you literally cannot do with grep on a mixed log file.