USE on Linux: CPU, memory, disk, network, and PSI

The USE resource checklist for Linux

What makes USE practical is that it gives you a finite sweep: for every resource, check exactly three cells and move on. When you are debugging an incident, that structure prevents both tunnel vision (checking only CPU) and paralysis (checking everything at once). In node_exporter + Prometheus, the key cells from Brendan Gregg’s original ~30-resource checklist are:

PSI: Pressure Stall Information

PSI was merged into Linux 4.20 (December 2018) by Johannes Weiner at Facebook, solving a long-standing observability gap: pre-PSI you had to infer pressure from load averages, vmstat’s r column, and iowait — none of which scale correctly across many cores or capture transient saturation.

PSI exposes at /proc/pressure/{cpu,memory,io} a six-value report per resource:

some avg10=5.20 avg60=3.11 avg300=1.02 total=1234567
full avg10=0.00 avg60=0.00 avg300=0.00 total=0

• some: at least one task was stalled waiting for this resource, averaged over 10s / 60s / 300s.
• full: all non-idle tasks were stalled. This is the critical signal — if full > 0 for memory or I/O for more than 10 seconds, you almost certainly have a crunch even if free-RAM looks plentiful (the kernel is thrashing reclaim).

PSI overhead: sub-1% (kernel-side counters, no per-task tracing).

node_exporter exposes PSI as node_pressure_cpu_waiting_seconds_total, node_pressure_memory_waiting_seconds_total, and node_pressure_io_waiting_seconds_total.

Why PSI beats load average

Load average mixes runnable and uninterruptible tasks, scales oddly with core count, and is a 1-minute exponential moving average — a transient 5-second CPU crunch barely registers. On a 64-core host, a load average of 10 is fine; on a 2-core host it is a disaster. PSI normalises by wall-clock time and is independent of core count.

Production alert thresholds:

• rate(node_pressure_cpu_waiting_seconds_total[5m]) > 0.1 (10% of time stalled on CPU) → warning
• rate(node_pressure_memory_stalled_seconds_total[5m]) > 0 for 10 min → memory crunch, page immediately
• rate(node_pressure_io_stalled_seconds_total[5m]) > 0 for 5 min → I/O crunch

Kubernetes 1.27+ uses PSI internally for eviction decisions for the same reason.

How USE signals look under failure

CPU-bound workload: Utilization climbs toward 100%, saturation (run-queue length) rises above the core count. This is the correct signal that scaling out will help. CPU per request stays flat — it is volume, not regression.

Memory leak: Utilization climbs slowly with si/so (swap-in / swap-out) eventually non-zero. PSI memory full goes from 0 to non-zero — the kernel is thrashing reclaim to free pages.

Disk-bound workload: %util near 100% on one or more disks, queue depth growing, PSI io some rising. This is what killed the Stripe worker above — visible in PSI, invisible in free-RAM.

Network-bound: TCP retransmits rising (node_netstat_Tcp_RetransSegs), NIC drops rising, bandwidth at negotiated maximum.