USE on Linux: CPU, memory, disk, network, and PSI

A Stripe webhook worker stalled for 11 minutes in 2024. Free-RAM dashboards showed plenty of available memory. The actual cause was kernel memory pressure at 90% — the kernel was thrashing reclaim. PSI caught the next incident in 30 seconds. Free-RAM dashboards still showed nothing.

The USE resource checklist for Linux

Brendan Gregg’s original USE checklist for Linux covers roughly 30 resource × U/S/E cells. In node_exporter + Prometheus, the key ones are:

PSI: Pressure Stall Information

PSI was merged into Linux 4.20 (December 2018) by Johannes Weiner at Facebook, solving a long-standing observability gap: pre-PSI you had to infer pressure from load averages, vmstat’s r column, and iowait — none of which scale correctly across many cores or capture transient saturation.

PSI exposes at /proc/pressure/{cpu,memory,io} a six-value report per resource:

some avg10=5.20 avg60=3.11 avg300=1.02 total=1234567
full avg10=0.00 avg60=0.00 avg300=0.00 total=0

• some: at least one task was stalled waiting for this resource, averaged over 10s / 60s / 300s.
• full: all non-idle tasks were stalled. This is the critical signal — if full > 0 for memory or I/O for more than 10 seconds, you almost certainly have a crunch even if free-RAM looks plentiful (the kernel is thrashing reclaim).

PSI overhead: sub-1% (kernel-side counters, no per-task tracing).

node_exporter exposes PSI as node_pressure_cpu_waiting_seconds_total, node_pressure_memory_waiting_seconds_total, and node_pressure_io_waiting_seconds_total.

Why PSI beats load average

Load average mixes runnable and uninterruptible tasks, scales oddly with core count, and is a 1-minute exponential moving average — a transient 5-second CPU crunch barely registers. On a 64-core host, a load average of 10 is fine; on a 2-core host it is a disaster. PSI normalises by wall-clock time and is independent of core count.

Production alert thresholds:

• rate(node_pressure_cpu_waiting_seconds_total[5m]) > 0.1 (10% of time stalled on CPU) → warning
• rate(node_pressure_memory_stalled_seconds_total[5m]) > 0 for 10 min → memory crunch, page immediately
• rate(node_pressure_io_stalled_seconds_total[5m]) > 0 for 5 min → I/O crunch

Kubernetes 1.27+ uses PSI internally for eviction decisions for the same reason.

How USE signals look under failure

CPU-bound workload: Utilization climbs toward 100%, saturation (run-queue length) rises above the core count. This is the correct signal that scaling out will help. CPU per request stays flat — it is volume, not regression.

Memory leak: Utilization climbs slowly with si/so (swap-in / swap-out) eventually non-zero. PSI memory full goes from 0 to non-zero — the kernel is thrashing reclaim to free pages.

Disk-bound workload: %util near 100% on one or more disks, queue depth growing, PSI io some rising. This is what killed the Stripe worker above — visible in PSI, invisible in free-RAM.

Network-bound: TCP retransmits rising (node_netstat_Tcp_RetransSegs), NIC drops rising, bandwidth at negotiated maximum.