High Memory Usage and the Linux OOM Killer: Safe Diagnosis and Recovery

High “used” RAM on Linux does not automatically mean memory exhaustion. The kernel uses idle RAM for page cache and can reclaim it for applications. A real incident appears when MemAvailable remains low, swapping becomes active, processes stall, or the kernel invokes the OOM Killer.

Understand used, free, and available

free -h
cat /proc/meminfo | head -20

In free, examine available before free. MemAvailable estimates memory available to new applications without swapping, including reclaimable page cache and slab. A large cache often shows efficient RAM use rather than a leak.

Persistent declining availability, swap traffic, latency, and killed workloads are meaningful signals. One snapshot is insufficient; collect a time series.

1. Confirm memory pressure

free -h
vmstat 1 10
cat /proc/pressure/memory
swapon --show

In vmstat, si and so report swap-in and swap-out. Used swap alone is not proof of current pressure; continuous traffic with latency is concerning. PSI in /proc/pressure/memory measures time tasks are stalled by memory contention. Sustained full pressure means all non-idle workloads are stalled and the system risks thrashing or OOM.

2. Find memory-consuming processes

ps -eo pid,ppid,user,comm,rss,vsz,%mem --sort=-rss | head -20
top
systemd-cgtop

RSS is resident memory, but summing it may count shared pages more than once. VSZ includes mapped address space and is not physical RAM consumption. Inspect a candidate:

cat /proc/PID/status
cat /proc/PID/smaps_rollup
pmap -x PID | tail -1

smaps_rollup distinguishes private, shared, anonymous, and file-backed memory. In containers or systemd, inspect cgroups because a cgroup OOM can occur while the host still has memory.

3. Separate valid load, cache, and leaks

• Valid load: memory follows traffic and declines after load drops or workers recycle.
• Page cache: Cached is high while MemAvailable remains healthy.
• Memory leak: private or anonymous RSS grows across load cycles without returning to baseline.
• Unbounded queue: backlog retains objects; fix throughput and backpressure.
• tmpfs/shmem: files in /dev/shm or tmpfs consume RAM.
• Kernel/slab: process RSS does not explain usage; inspect Slab, SReclaimable, and relevant subsystems.

Graph RSS, request rate, queue depth, and deployment times together. Time correlation is more useful than restarting and erasing evidence.

4. Confirm an OOM kill

journalctl -k -g 'Out of memory|Killed process|oom-kill' --since today
dmesg -T | grep -Ei 'out of memory|killed process|oom-kill'
journalctl -u systemd-oomd --since today
oomctl

The kernel OOM Killer selects a task using heuristics, memory and swap consumption, and oom_score_adj. systemd-oomd is a userspace mechanism that may act earlier using cgroup and pressure data. Logs reveal whether the event was global or cgroup-scoped, the victim, usage, and constraints.

The killed process is not necessarily the only cause. Traffic spikes, incorrect limits, worker counts, another leak, or host overcommit may have created the condition.

5. Stabilize an active incident

1. Stop or reduce noncritical traffic and jobs.
2. Capture free, vmstat, process, cgroup, and OOM evidence.
3. Scale workloads elsewhere or reduce concurrency safely.
4. Restart the leaking service when immediate recovery is required.
5. Add temporary swap only with a clear latency and storage assessment.
6. Monitor afterward; a restart is not a root-cause fix.

Do not treat echo 3 > /proc/sys/vm/drop_caches as a RAM fix. Dropping useful cache can worsen I/O and latency; the kernel already reclaims cache when required.

6. Swap is a buffer, not free RAM

Swap can absorb bursts and move cold pages out of RAM, buying response time, but storage is far slower. No swap creates a thinner safety margin; large swap does not repair an infinite leak.

swapon --show
cat /proc/sys/vm/swappiness
vmstat 1

Tune vm.swappiness only after measuring the workload. A low value does not mean “never swap.” Databases, latency-sensitive services, and Kubernetes require workload-specific guidance.

7. Set limits and preserve headroom

A systemd service can use cgroup controls:

[Service]
MemoryHigh=1500M
MemoryMax=2G
OOMPolicy=stop

MemoryHigh creates soft pressure and throttling; MemoryMax is a hard ceiling. Derive values from valid peak data. After editing the unit:

sudo systemctl daemon-reload
sudo systemctl restart example.service
systemctl show example.service -p MemoryCurrent -p MemoryPeak -p MemoryHigh -p MemoryMax

Container limits must include heap, native allocations, page cache, and sidecars. Reserve headroom for the kernel, SSH, monitoring, and background daemons; do not allocate 100% of host RAM.

8. Fix the application instead of only adding RAM

• Limit workers and concurrency using peak memory per worker.
• Stream or chunk data instead of loading entire files or query results.
• Bound queues, timeouts, payloads, and uploads.
• Configure cache eviction and maximum memory.
• Recycle workers after a measured request count as risk reduction, not a permanent leak mask.
• Profile heap and native memory in a production-like environment.

More RAM is valid when the working set truly grows. If the curve grows without bound, it only postpones OOM.

9. Monitor and alert early

Track MemAvailable, cgroup working set, swap I/O, memory PSI, OOM events, restart counts, and queue depth. Alert on sustained pressure and duration rather than RAM-used percentage alone.

Retain post-deployment metrics long enough to detect slow leaks. Alerts should link to a runbook with evidence commands, service ownership, and explicit scale or restart conditions.

Common mistakes

• Calling high used RAM a leak without checking available and cache.
• Adding all RSS values as an exact system total.
• Restarting before preserving logs and charts.
• Disabling OOM behavior or setting oom_score_adj=-1000 broadly.
• Running drop_caches periodically as optimization.
• Adding swap or RAM while queues and concurrency remain unbounded.
• Monitoring only the host and missing container/cgroup OOM events.

Incident checklist

1. Confirm MemAvailable, swap activity, and PSI over time.
2. Identify the process/cgroup and memory type that is growing.
3. Read kernel and systemd-oomd logs to classify the OOM.
4. Preserve evidence before restart or termination.
5. Shed load and recover services by priority.
6. Fix leaks, concurrency, queue behavior, or cache policy.
7. Set appropriate MemoryHigh/MemoryMax and host headroom.
8. Load-test and alert before the OOM threshold.

Conclusion

Linux memory investigation begins with pressure and reclaimability, not a red used-memory number. Combine MemAvailable, PSI, swap I/O, process and cgroup metrics, and OOM logs to find the cause. A restart may restore service, but limits, backpressure, and application fixes prevent recurrence.

References

• Linux Kernel: The /proc filesystem
• Linux Kernel: Pressure Stall Information
• Linux Kernel: VM sysctl documentation
• systemd: Resource control
• systemd: systemd-oomd