Signals — deeper

Signals are asynchronous notifications — the kernel interrupts the process and delivers them.

A signal is a small integer sent by the kernel (or by another process with appropriate permissions) to a target process. The kernel interrupts the target at an instruction boundary and calls the registered signal handler. If no custom handler is installed, the default action runs (which for most signals is: terminate the process).

# List all signals and their default actions:
kill -l
# 1) SIGHUP    2) SIGINT    3) SIGQUIT   4) SIGILL    5) SIGTRAP
# 6) SIGABRT   7) SIGBUS    8) SIGFPE    9) SIGKILL  10) SIGUSR1
# 11) SIGSEGV 12) SIGUSR2  13) SIGPIPE  14) SIGALRM  15) SIGTERM
# 18) SIGCONT  19) SIGSTOP  20) SIGTSTP ...

# Signal by number or name — both are equivalent:
kill -15 1234      # SIGTERM by number
kill -TERM 1234    # SIGTERM by name
kill -SIGTERM 1234 # SIGTERM with SIG prefix

# Default actions (if no handler is installed):
# SIGTERM (15) → terminate
# SIGINT  (2)  → terminate (from Ctrl-C)
# SIGHUP  (1)  → terminate (but daemons often handle it as "reload")
# SIGQUIT (3)  → terminate + core dump
# SIGKILL (9)  → terminate (CANNOT be caught/ignored — kernel-enforced)
# SIGSTOP (19) → stop (CANNOT be caught/ignored — kernel-enforced)
# SIGCONT (18) → continue (resumes a stopped process)

SIGTERM vs SIGKILL: the right tool for the right job.

# SIGTERM (15): polite shutdown request
# The process CAN:
# - Catch it and run cleanup code
# - Flush write buffers, close connections, delete temp files
# - Ignore it (badly written processes sometimes do this)
# - Take time before actually exiting
#
# The operator pattern: send SIGTERM, wait, then SIGKILL if needed:
kill -TERM 1234
sleep 10
kill -0 1234 2>/dev/null && kill -KILL 1234
# kill -0 doesn't send a signal — it just checks if the process exists
# If it still exists after 10 seconds, escalate to SIGKILL

# SIGKILL (9): forcible termination — no exceptions
# The kernel forcibly removes the process from the scheduler.
# The process's signal handlers are NEVER invoked.
# No cleanup, no flush, no graceful shutdown.
# Consequences of kill -9:
# - Open database connections left dangling
# - Write-ahead logs not flushed (requires crash recovery on restart)
# - Temp files left on disk
# - systemd restart may be needed since the process did not acknowledge shutdown
kill -KILL 1234    # or: kill -9 1234

# SIGHUP (1): originally "hangup" (terminal disconnected)
# Daemons conventionally handle SIGHUP as "reload configuration"
# This lets you update nginx.conf and reload without dropping connections:
sudo kill -HUP $(cat /var/run/nginx.pid)
# nginx re-reads its config, gracefully drains old workers, starts new ones

# Or with systemctl:
sudo systemctl reload nginx   # sends SIGHUP internally

Why D-state processes ignore even SIGKILL.

SIGKILL is the one signal the kernel guarantees cannot be ignored — except when the process is in D (uninterruptible sleep) state. In D state, the process is inside a non-preemptible kernel operation, typically waiting for:

• Block device I/O (a read from a disk that has stopped responding)
• An NFS operation on a mounted filesystem that went away
• A kernel lock (very briefly — usually microseconds, not minutes)

# Identify D-state processes:
ps aux | awk '$8 ~ /D/'

# If a process is stuck in D state for minutes, the problem is the subsystem:
# 1. Check for hung mounts:
mount | grep nfs
df -h  # if df hangs, an NFS mount is unresponsive

# 2. Check kernel logs for I/O errors:
sudo dmesg | tail -30 | grep -E 'error|timeout|hung|reset'
# "task ... blocked for more than 120 seconds" → stuck I/O

# 3. The fix is to resolve the I/O issue (unmount the NFS, replace the disk),
#    NOT to send more signals.
#
# The reason SIGKILL cannot reach D-state processes:
# The kernel delivers signals at safe points — when returning from a syscall
# or kernel preemption point. In uninterruptible sleep, the process is INSIDE
# a critical kernel section. Delivering a signal would leave kernel data structures
# in an inconsistent state. Safety over liveness: the kernel accepts a zombie
# stuck in D over a kernel panic.

Process groups and sessions: why Ctrl-C kills a whole pipeline.

Processes are organized into process groups (a set of related processes, usually a pipeline) and sessions (all processes started from a terminal login). The terminal sends signals to the foreground process group — not to a single process.

# Start a two-stage pipeline:
sleep 300 | cat -

# In another terminal, find the process group:
ps -o pid,pgid,sid,comm | grep -E 'sleep|cat'
# PID   PGID   SID    COMM
# 5001  5001   4999   sleep    ← pgid=pid: sleep is process group leader
# 5002  5001   4999   cat      ← same pgid: same group as sleep

# PGID 5001 is the foreground process group of the terminal.
# Ctrl-C sends SIGINT to ALL processes in PGID 5001 — both sleep AND cat.
# That is why the whole pipeline dies, not just the first stage.

# Sessions:
# A session (SID) groups all processes from one terminal login.
# The session leader is the shell. When the terminal closes, SIGHUP is sent
# to the foreground process group. This is why backgrounded processes
# die when you close an SSH session — they receive SIGHUP.
# 'nohup' or 'disown' detaches a process from the session's SIGHUP:
nohup long_running_job &   # ignores SIGHUP from terminal close