Linux Memory Pressure and OOM

Linux memory incidents are usually not “free memory is low.” The kernel uses memory for page cache, slab, anonymous pages, file mappings, buffers, kernel stacks, and cgroups. Healthy systems often keep free memory low because unused RAM is wasted RAM.

The operational question is whether reclaim, compaction, swapping, cgroup limits, Pressure Stall Information, or the OOM killer are affecting workloads.

Command Examples

free -h
cat /proc/meminfo
cat /proc/pressure/memory
vmstat 1
ps -eo pid,ppid,comm,rss,vsz,%mem --sort=-rss | head
journalctl -k -g 'Out of memory|Killed process|oom-kill'

Example output and meaning:

Start with system pressure, then separate process RSS, kernel memory, page cache, swap activity, and cgroup limits.

Memory Types

available memory is usually a better quick signal than free memory because it estimates memory that can be used without heavy reclaim.

Page Cache and Reclaim

Linux aggressively caches file data. Page cache is useful until reclaim has to fight active workloads.

Useful checks:

grep -E 'MemAvailable|Cached|Dirty|Writeback|Slab|SReclaimable|SUnreclaim' /proc/meminfo
sar -B 1
vmstat 1

Watch for sustained page scanning, dirty writeback stalls, and low MemAvailable. Dropping caches is rarely a fix; it can hide evidence and make the next read path slower.

OOM Killer

The OOM killer chooses a victim when the kernel cannot satisfy memory allocation after reclaim. The decision uses badness scoring, memory use, privileges, and oom_score_adj.

Inspect scores:

cat /proc/<pid>/oom_score
cat /proc/<pid>/oom_score_adj
grep -E 'VmRSS|VmSize|RssAnon|RssFile|RssShmem' /proc/<pid>/status

In containers, the kernel can kill a process because the cgroup limit is hit even when the host has memory available.

Cgroups and Containers

For cgroup v2:

cat /sys/fs/cgroup/memory.current
cat /sys/fs/cgroup/memory.max
cat /sys/fs/cgroup/memory.high
cat /sys/fs/cgroup/memory.events
cat /sys/fs/cgroup/memory.pressure

Important events:

Kubernetes memory limits map to cgroup limits. A pod can be OOMKilled while the node looks mostly healthy.

OOM Comparison Matrix

Do not assume every OOMKilled is a node outage. The first split is host boundary, cgroup boundary, kubelet eviction, or language runtime.

Runtime Memory Examples

Language runtimes reserve and report memory differently, so a single RSS number does not explain the whole failure.

Example: a Java container with memory.max=1GiB and -Xmx1g can still die because thread stacks, direct buffers, metaspace, and libc allocations need memory outside the Java heap. Leave headroom or use container-aware heap sizing.

Example: a Go service can show a stable application heap in pprof while RSS grows from mmap, Cgo, or retained spans. Compare pprof with /proc/<pid>/smaps_rollup before blaming the garbage collector.

THP, NUMA, and Swap

Transparent Huge Pages can improve TLB efficiency for some workloads and hurt latency for others through compaction and allocation stalls. NUMA can make memory access slower when processes run far from their memory. Swap can preserve availability but create latency spikes when hot pages are swapped out.

Checks:

cat /sys/kernel/mm/transparent_hugepage/enabled
numastat
numastat -p <pid>
swapon --show
cat /proc/swaps

Treat THP and swap policy as workload-specific. Databases often need explicit guidance from vendor docs and measured testing.

Leak Triage

1. Confirm whether RSS is growing or only VSZ is large.
2. Compare process memory, cgroup memory, and node memory.
3. Split anonymous RSS from file-backed RSS.
4. Check whether page cache, slab, or dirty writeback dominates.
5. Review deployment changes, traffic changes, and batch jobs.
6. Capture heap profiles when the application runtime supports them.
7. If kernel memory grows, inspect slab caches and subsystem counters.

Useful commands:

pmap -x <pid> | tail -20
grep -E 'Rss|Pss|Private|Shared' /proc/<pid>/smaps_rollup
slabtop

Runbook

1. Confirm whether the symptom is latency, allocation failure, swap storm, cgroup OOM, or node OOM.
2. Save free -h, /proc/meminfo, PSI, vmstat, top RSS processes, and kernel OOM logs.
3. Check cgroup memory files for affected services or containers.
4. Identify dominant memory: anonymous RSS, page cache, slab, dirty pages, or swap.
5. Apply the smallest mitigation: reduce concurrency, restart one leaking workload, raise a cgroup limit, disable a batch job, or shed traffic.
6. After recovery, add alerts for PSI, cgroup oom_kill, swap activity, and sustained RSS growth.

Study Cards

References

• Linux proc meminfo documentation
• Linux cgroup v2 documentation
• Linux PSI documentation
• proc_pid_oom_score_adj(5)