Memory, the Page Cache & the OOM Killer

In this module, you'll learn to:

• Read free correctly, and explain the page cache and why it isn't "used" memory
• Find the processes using the most memory, and understand swap and memory limits
• Recognise an out-of-memory kill and find the evidence the process itself never logs

Before you start: finish Processes, Signals & File Descriptors. Open a shell on your lab VM with multipass shell lab.

Unused memory is wasted memory

Reading from disk is thousands of times slower than reading from memory. So Linux keeps recently read files in otherwise idle memory, in the page cache. The next read comes from memory instead of disk. When a program needs that memory, the kernel simply drops some cached pages and hands it over.

That's why a healthy server that has been up for a while shows very little "free" memory. It isn't a leak. The kernel is putting idle memory to work.

free -h
#                total        used        free      shared  buff/cache   available
# Mem:           1.9Gi       310Mi       1.1Gi       1.2Mi       600Mi       1.6Gi

Alert on available running low, never on free.

Watch the page cache work

dd if=/dev/urandom of=~/big.bin bs=1M count=500   # a 500 MB file

sync; echo 3 | sudo tee /proc/sys/vm/drop_caches  # empty the cache (for learning only)
free -h                                           # note buff/cache

time cat ~/big.bin > /dev/null                    # first read: from disk
free -h                                           # buff/cache grew by about 500 MB

time cat ~/big.bin > /dev/null                    # second read: from memory, much faster
rm ~/big.bin

buff/cache grew, available barely moved, and the second read was far faster. Never drop caches on a production server to "free memory": it only makes the next reads slow.

Who is using the memory

ps aux --sort=-rss | head -6     # the biggest processes by resident memory
cat /proc/meminfo | head -20     # the kernel's full breakdown
vmstat 1 5                       # memory, swap in/out (si/so), and CPU, every second

ps shows two sizes for each process. VSZ (virtual size) is everything the process has mapped, much of which may never be used. RSS (resident set size) is what's actually in memory right now. RSS is the one to compare. Shared libraries are counted in every process that uses them, so adding up RSS across processes overstates the total.

A safety net, not extra memory

Swap is disk space the kernel uses to hold memory pages nobody has touched recently. It lets a server survive a short spike, but a server that's actively swapping (non-zero si and so columns in vmstat, all the time) is short of memory and will be very slow.

swapon --show        # nothing on most cloud images: no swap by default

Many cloud images ship with no swap at all, and Kubernetes nodes traditionally run without it. Then nothing slows down gradually: when memory runs out, the next step is the OOM killer.

cgroups and the OOM killer

When the system, or a group of processes, runs out of memory and nothing more can be reclaimed, the kernel's OOM killer (out of memory) picks a process and kills it with SIGKILL. It favours the process using the most memory, adjusted by its oom_score_adj setting. The victim gets no warning and no chance to log anything.

cat /proc/$$/oom_score          # how likely your shell is to be chosen
cat /proc/$$/oom_score_adj      # the adjustment, from -1000 (never) to 1000

Limits can also apply to a group of processes. systemd puts every service in its own control group (cgroup), and MemoryMax= in a unit caps that service's memory. Hitting the cap triggers an OOM kill inside the group only, which protects the rest of the server. Container memory limits work exactly the same way. Kubernetes reports the result as OOMKilled with exit code 137 (128 + 9, for SIGKILL), which the Kubernetes track's Scaling module builds on.

The report that never finishes

A nightly report job runs as a service with a memory limit. Its log says it started, then nothing: no error, no "finished". Run it the way the scheduler does:

sudo tee /usr/local/bin/reportgen.py <<'EOF'
import time
print("report generation started", flush=True)
rows = []
for i in range(80):
    rows.append(b"x" * 10 * 1024 * 1024)   # 10 MB more each step
    time.sleep(0.1)
print("report finished", flush=True)
EOF

sudo systemd-run --unit=reportgen -p MemoryMax=150M -p MemorySwapMax=0 \
  /usr/bin/python3 /usr/local/bin/reportgen.py
sleep 10
journalctl -u reportgen --output=cat | grep -v "^Started\|^reportgen"

The last line shows only what the program itself printed. Your job: prove what happened to it, and make the job finish. Try it before opening a hint.

Key terms

• Page cache: recently used file data kept in memory, given back when programs need it.
• available: the memory a new workload could use without swapping. The number to watch.
• RSS and VSZ: memory actually in RAM, and everything a process has mapped.
• Swap: disk space for memory pages nobody is using; constant swapping means a memory shortage.
• cgroup: a group of processes the kernel limits as one, used for every service and container.
• OOM killer: the kernel mechanism that kills a process with SIGKILL when memory runs out.
• Exit code 137: a process killed by SIGKILL, usually by the OOM killer.

Pop quiz

Five questions: three on the ideas in this module, and two scenarios where you apply them. The order changes every time you take it, and 4 out of 5 passes.