High Linux Load and Slow Servers: Diagnose CPU, I/O, Memory, and Processes

A load average of 20 does not automatically mean 100% CPU usage. Linux load includes tasks running or waiting for CPU and tasks in uninterruptible sleep, often waiting for I/O. A high-load server may be constrained by CPU, disks, NFS, memory reclaim, a hypervisor, or another kernel path.

This guide moves from system-wide signals to the exact process or cgroup causing pressure with uptime, top, vmstat, pidstat, iostat, sar, and Pressure Stall Information (PSI).

First, define what users experience

Attach “slow” to a measurable symptom: rising API latency, timeouts, lagging SSH, one slow service, reduced throughput, or impact limited to a container or tenant. Record the timestamp and timezone, and correlate with deployments, backups, cron, and traffic.

Do not restart immediately. It removes process state, queues, and evidence. When data loss or cascading failure is active, shed load first but preserve a minimal snapshot.

1. Interpret load average correctly

uptime
cat /proc/loadavg
nproc
lscpu

The three load values cover roughly 1, 5, and 15 minutes. They count runnable tasks (R) and tasks in uninterruptible I/O wait (D); they are not CPU percentages.

Comparing load with logical CPU count is only an orientation. Load 8 on eight CPUs may be healthy for a CPU-bound workload meeting its SLA; load 4 can be severe when every request waits on one slow disk. Trend matters: a one-minute value above longer windows means rising pressure, below them indicates recovery, and all three elevated indicates sustained pressure.

2. Take a top-level snapshot

top
top -H -p PID

• us: user-space work.
• sy: kernel/system work.
• id: idle time.
• wa: CPU idle while I/O is outstanding; it is not disk utilization.
• hi/si: hardware/software interrupts.
• st: vCPU time taken by the hypervisor.

Press 1 for per-CPU view, P for CPU sorting, and M for memory. A process at 400% can legitimately consume four logical CPUs. Because snapshots miss bursts, continue with interval sampling.

3. Use vmstat to separate CPU, blocked tasks, and swapping

vmstat 1 10
vmstat -y 1 10

-y skips the first line, which averages since boot.

• r: runnable tasks; sustained values above CPU count suggest a run queue.
• b: tasks blocked on I/O.
• si/so: swap activity.
• us/sy/id/wa/st: CPU distribution.
• cs: context switches, meaningful alongside throughput and scheduler load.

High r with near-zero idle and low wait points toward CPU saturation. High b and wait points toward I/O. Continuous swap activity suggests memory pressure. High steal time indicates hypervisor contention.

4. Attribute pressure with pidstat

pidstat -u -r -d -w -p ALL 1 10
pidstat -t -u -p PID 1 10

-u reports CPU, -r memory and faults, -d I/O, -w context switches, and -t threads. Thread view matters for JVMs, databases, web servers, and thread pools where one hot thread can hide inside a process total.

ps -eo pid,ppid,stat,psr,ni,%cpu,%mem,wchan:24,comm,args \
  --sort=-%cpu | head -30

5. Find tasks in uninterruptible sleep

Important states include R runnable, S interruptible sleep, D uninterruptible sleep, Z zombie, and T stopped. Many D-state tasks can raise load while CPU remains idle.

ps -eo state,pid,ppid,wchan:32,comm,args | awk '$1 ~ /^D/'
cat /proc/PID/stack
cat /proc/PID/wchan

The wait channel or kernel stack may expose block I/O, NFS, filesystem, locking, or driver waits. A D-state task usually cannot be killed immediately because signals are handled only when the kernel operation returns. Reboot may be a last resort, but identify the failing path first.

6. Identify the constrained block device

iostat -xz 1 10

• r/s, w/s: IOPS.
• rkB/s, wkB/s: throughput.
• await: average queue plus service latency.
• aqu-sz: average queue depth.
• %util: time the device reports busy.

Do not apply one “100% util” rule to every device. NVMe, RAID, SAN, and virtual disks support different concurrency. Judge latency against the workload SLA and its baseline, along with queue and throughput.

dmesg -T | grep -Ei 'error|timeout|reset|nvme|blk|ext4|xfs'
journalctl -k --since '-30 min'

7. Distinguish memory usage from pressure

free -h
vmstat 1
cat /proc/meminfo
cat /proc/pressure/memory

Linux uses otherwise idle RAM for page cache, so low free is not automatically bad. Inspect available, swap activity, reclaim behavior, and PSI. Tasks can stall on reclaim, compaction, or swap before an OOM event.

ps -eo pid,ppid,%mem,rss,vsz,comm,args --sort=-rss | head -30
systemd-cgtop
cat /sys/fs/cgroup/system.slice/SERVICE.service/memory.events

Do not schedule periodic cache dropping as a fix. It can increase I/O and hide the actual cause.

8. Use PSI to measure lost work time

cat /proc/pressure/cpu
cat /proc/pressure/memory
cat /proc/pressure/io

PSI reports the percentage of time tasks were delayed by CPU, memory, or I/O scarcity over 10, 60, and 300 seconds. some means at least one task was stalled; full means all non-idle tasks were stalled together. Sustained full memory or I/O pressure is especially serious.

Load tells how many tasks are runnable or blocked; PSI tells how much productive time contention took away. With cgroup v2, per-service cpu.pressure, memory.pressure, and io.pressure prevent one workload from disappearing inside host averages.

9. Classify CPU saturation

When user time dominates, profile functions rather than simply killing the process:

perf top
perf stat -p PID sleep 10

High system time can come from syscalls, networking, filesystems, locks, or kernel work. High softirq requires packet and NIC investigation:

mpstat -P ALL 1 10
mpstat -I ALL 1 10
cat /proc/interrupts

High steal time on a VM means the hypervisor is not scheduling enough vCPU. Guest code optimization cannot resolve a noisy host; change instance/host or contact the provider.

10. Examine run queues and context switches

mpstat -P ALL 1
pidstat -w -p ALL 1
sar -q 1 10

Full CPU usage is not necessarily bad when throughput and latency meet targets. A growing run queue and CPU wait are the concern. Thread explosion, lock contention, or spin loops can produce high context switching with poor throughput. Compare worker count, thread pools, connection pools, garbage collection, and CPU quotas with the last change.

11. Containers and cgroups

A container can be throttled while the host has spare resources:

docker stats --no-stream
systemd-cgtop
cat /sys/fs/cgroup/CGROUP/cpu.stat
cat /sys/fs/cgroup/CGROUP/memory.current
cat /sys/fs/cgroup/CGROUP/memory.events

Rising nr_throttled and throttled_usec in cpu.stat indicate CPU quota pressure. In Kubernetes, inspect pod limits, throttling, and node pressure. Do not raise limits before understanding traffic and regression.

12. Historical evidence beats a snapshot

sar -u -f /var/log/sysstat/sa19
sar -q -f /var/log/sysstat/sa19
sar -r -f /var/log/sysstat/sa19
sar -d -p -f /var/log/sysstat/sa19

When the incident has ended, current top cannot reconstruct it. Enable sysstat or a monitoring agent in advance. Correlate with deployments, cron, backups, traffic, database checkpoints, and cloud events. Compare the same hour and weekday to distinguish expected peaks from regressions.

13. Mitigate safely during an incident

• Rate-limit or shed noncritical requests.
• Pause batch jobs, backups, index builds, or low-priority workers.
• Scale after confirming saturation and application readiness.
• Reduce concurrency when a dependency is overloaded; more workers may worsen it.
• Roll back the deployment or configuration that caused regression.
• Use renice/ionice cautiously for batch work, not as the permanent fix.

Do not kill a database, flush caches, drop caches, or reboot solely because load is high. Capture metrics, process lists, stacks, logs, and I/O state first, and assess data and failover impact.

14. Common false conclusions

• High load always means high CPU: D-state tasks count too.
• Low free memory means shortage: page cache is useful.
• High iowait proves a failed disk: inspect devices, tasks, and the storage path.
• 100% util means every disk is saturated: concurrent devices differ.
• A 300% process is a bug: it may legitimately use three cores.
• More workers always increase throughput: contention and queues may grow.
• A restart is a fix: it often erases symptoms and evidence.

Fifteen-minute playbook

1. Record latency, errors, throughput, and time.
2. uptime, nproc: trend and CPU count.
3. vmstat -y 1 10: runnable, blocked, swap, wait, steal.
4. cat /proc/pressure/*: which resource stalls tasks.
5. pidstat -u -r -d -w -p ALL 1 10: responsible tasks.
6. iostat -xz 1 10: device latency, queue, throughput.
7. ps: D-state tasks, hot threads, command lines.
8. Inspect cgroup quotas and throttling.
9. Correlate logs, deployments, cron, and traffic.
10. Apply reversible load reduction, verify the SLA, and write a postmortem.

Conclusion

Load average is the starting signal, not a diagnosis. Understanding a slow Linux host requires separating CPU wait from I/O wait, sampling trends, attributing work to processes, threads, and cgroups, and measuring lost time with PSI. Combining vmstat, pidstat, iostat, and historical data turns “high load” into a specific cause, a reversible mitigation, and a verifiable prevention change.

References

• Linux man-pages: /proc/loadavg
• procps-ng: vmstat
• sysstat: pidstat
• Linux Kernel: Pressure Stall Information