Processes, Signals & File Descriptors

In this module, you'll learn to:

• Inspect processes with ps, top, and /proc, and read their states
• Stop processes with the right signal, and explain zombies and stuck D-state processes
• Find a process's open files and limits, and raise a service's file descriptor limit correctly

Before you start: finish systemd: Services, Timers & Logs. Open a shell on your lab VM with multipass shell lab.

A process is a running program

When you run a program, the kernel creates a process: a copy of the program in memory with its own ID (the PID), the user it runs as, its current directory, its environment variables, and its open files. Every process has a parent (its PPID) that started it, all the way back to systemd at PID 1.

ps aux | head                   # every process: user, PID, CPU %, memory %, state, command
ps -ef --forest | less          # the parent and child tree
pstree -p | head -20            # the same tree, more compact
top                             # live view; press P to sort by CPU, M by memory, q to quit
pgrep -a sshd                   # PIDs and commands matching a name

The kernel shows each process as a directory under /proc/<PID>, and the commands above read from there. You can read it directly too:

echo $$                              # the PID of your own shell
cat /proc/$$/status | head -10       # name, state, PID, parent, user
tr '\0' ' ' < /proc/$$/cmdline; echo # the exact command line
ls -l /proc/$$/cwd                   # its current directory

R, S, D, Z, and T

The STAT column in ps aux shows what each process is doing. Most of the time almost everything is sleeping, which is normal.

Load average (from uptime or top) counts processes that are running, waiting for CPU, or in D state. So a high load with an idle CPU usually means processes are stuck waiting on storage, not short of CPU. Part 3 uses this to diagnose slow servers.

Try it · Make a zombie. This parent starts a child that exits immediately, but never collects it:

python3 -c 'import os, time
if os.fork() == 0:
    os._exit(0)
time.sleep(300)' &

ps -o pid,ppid,stat,cmd --ppid $!     # the child shows Z and <defunct>
kill $!                               # end the parent...
ps -o pid,ppid,stat,cmd --ppid $!     # ...and the zombie is gone

You can't kill a zombie, because it's already dead. When its parent exits, systemd adopts the zombie and collects it. The fix for a zombie build-up is always the parent.

Asking, and telling, a process to stop

A signal is a short message from the kernel to a process. kill sends one, and despite its name, most signals aren't fatal.

sleep 600 &
kill $!            # SIGTERM: the polite request
sleep 600 &
kill -9 $!         # SIGKILL: only when TERM didn't work
pkill -f "sleep 600"   # by matching the command line

Always try SIGTERM first and give it a few seconds. SIGKILL can leave half-written files, stale lock files, and lost data. systemctl stop does exactly this for you: TERM, a timeout, then KILL.

nice and renice set a process's priority, from -20 (most favoured) to 19 (least). nice -n 10 tar czf backup.tgz /data runs a backup without slowing everything else down.

Every open file is a file descriptor

Each file, network connection, or pipe a process has open is a file descriptor (fd), a small number. Every process starts with three: 0 is stdin, 1 is stdout, and 2 is stderr.

ls -l /proc/$$/fd                     # your shell's open file descriptors
sudo apt install -y lsof
sudo lsof -p "$(pgrep -o sshd)"       # everything one process has open
sudo lsof -i :22                      # which process has port 22 open

The kernel limits how many descriptors one process may hold. A busy web server or database can need tens of thousands, one per connection.

ulimit -n                         # your shell's limit (the soft limit)
ulimit -Hn                        # the ceiling it may be raised to (the hard limit)
cat /proc/$$/limits               # every limit for a running process

That split catches people out: they raise the limit in limits.conf, sign in again, see a big number from ulimit -n, and the service still fails, because services never read that file.

Too many open files

A new service, filehog, starts and then dies a few seconds later, every time. Create it:

sudo tee /usr/local/bin/filehog.py <<'EOF'
import time
files = []
for i in range(200):
    files.append(open("/etc/hostname"))
    time.sleep(0.02)
print(f"opened {len(files)} files, now serving", flush=True)
time.sleep(3600)
EOF

sudo tee /etc/systemd/system/filehog.service <<'EOF'
[Unit]
Description=A service that needs a lot of open files

[Service]
ExecStart=/usr/bin/python3 /usr/local/bin/filehog.py
LimitNOFILE=64
EOF

sudo systemctl daemon-reload
sudo systemctl start filehog
sleep 5; systemctl status filehog

Your job: find out why it dies, and give it the limit it needs, the way you would for a real service. Try it before opening a hint.

Key terms

• PID and PPID: a process's ID, and its parent's ID.
• /proc/<PID>: the kernel's live view of a process: status, command line, limits, open files.
• D state: waiting on I/O and unkillable until it finishes. Counted in the load average.
• Zombie: a finished process whose parent hasn't collected its exit status.
• SIGTERM and SIGKILL: the polite request to stop, and the forced stop with no clean-up.
• File descriptor: the number for each open file, socket, or pipe in a process.
• LimitNOFILE: the open-file limit for a systemd service.

Pop quiz

Five questions: three on the ideas in this module, and two scenarios where you apply them. The order changes every time you take it, and 4 out of 5 passes.