Seeing Inside a Process: strace, lsof & eBPF

In this module, you'll learn to:

• Explain system calls, and trace them with strace to see what a program is really doing
• Find a process's files, sockets, and connections with lsof and ss
• Use eBPF tools to watch every new process, file open, or connection on the system

Before you start: finish A Troubleshooting Method. Open a shell on your lab VM with multipass shell lab, and install the tools:

sudo apt install -y strace lsof

Every program talks to the kernel

A program can't touch a file, a network connection, or another process by itself. It asks the kernel, through system calls: openat to open a file, read and write to move data, connect to open a connection, execve to start a program. Watching those calls shows what a program is doing, even when it prints nothing and you don't have its source code.

What is this process doing?

strace ls /tmp                          # every system call ls makes (to stderr)
strace -e trace=openat cat /etc/hostname     # only file opens
strace -c ls /tmp                       # a summary: calls, counts, time spent
strace -f -tt -o /tmp/trace.txt cmd     # follow child processes, timestamps, save to a file
sudo strace -p <PID>                    # attach to a process that's already running

The output reads as call(arguments) = result. A result of -1 comes with an error name, such as ENOENT (no such file) or EACCES (permission denied), which is often the whole answer.

Try it · Which files does a program really read? This shows exactly which configuration a command loads, which is the quickest way to settle "is it even reading my config?":

strace -f -e trace=openat curl -s -o /dev/null https://example.com 2>&1 \
  | grep -v ENOENT | grep -E "/etc|ssl"

You'll see curl open /etc/nsswitch.conf, /etc/hosts, and /etc/resolv.conf, the lookup path from DNS, Firewalls & Packet Capture, and then the certificate store under /etc/ssl.

strace slows the traced process down a lot. Use it on production processes briefly, for a specific question, and detach (Ctrl+C) as soon as you have the answer.

Files and connections

lsof lists open files, and on Linux almost everything is a file, including network connections:

sudo lsof -p <PID>                  # everything one process has open
sudo lsof /path/to/file             # which processes have this file open
sudo lsof +L1                       # deleted files still held open
sudo lsof -i :443                   # what's using port 443
sudo lsof -i -a -p <PID>            # just this process's network connections
sudo ss -tnp | grep <PID>           # the same connections, from ss

Safe tracing for the whole system

eBPF lets small, verified programs run inside the kernel, attached to events such as "a process started" or "a file was opened". The kernel checks each program before it runs, so it can't crash the system, and the overhead is low enough for production. Many modern monitoring, networking, and security tools are built on it.

The bcc tools are ready-made eBPF programs for common questions. On Ubuntu they end in -bpfcc:

sudo apt install -y bpfcc-tools linux-headers-$(uname -r)

sudo execsnoop-bpfcc         # every new process, as it starts, with its arguments
sudo opensnoop-bpfcc         # every file opened, by any process
sudo tcpconnect-bpfcc        # every outgoing TCP connection
sudo biolatency-bpfcc 5 1    # how long disk I/O takes, as a histogram

execsnoop is especially useful. Leave it running for a minute on a busy server and you often find a cron job or health check starting thousands of short-lived processes that never show up in top.

bpftrace is a small language for writing your own one-line traces:

sudo apt install -y bpftrace
# Every file opened, and which program opened it
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s %s\n", comm, str(args->filename)); }'

For CPU questions ("where does this process spend its time?"), perf top shows the hottest functions live, and perf record -g followed by a flame graph shows the whole picture. That's worth learning once you're comfortable with the tools above.

Running, but doing nothing

The orders-sync service should sync orders with the inventory API every few seconds. It's "running" according to systemd, uses no CPU, and has logged nothing for ten minutes. Start both services:

sudo systemd-run --unit=inventory-api nc -lk 127.0.0.1 9000

sudo tee /usr/local/bin/orders_sync.py <<'EOF'
import socket
print("syncing orders with inventory-api", flush=True)
s = socket.create_connection(("127.0.0.1", 9000))
s.sendall(b"GET /stock HTTP/1.0\r\n\r\n")
reply = s.recv(4096)
print("got reply:", reply[:40], flush=True)
EOF

sudo systemd-run --unit=orders-sync python3 /usr/local/bin/orders_sync.py
sleep 3; systemctl status orders-sync --no-pager

Your job: find out what orders-sync is stuck on, without reading its source, and get it moving. Try it before opening a hint.

Key terms

• System call: a program's request to the kernel, such as openat, read, connect, or execve.
• strace: traces one process's system calls; slow, so use it briefly.
• ENOENT, EACCES: "no such file" and "permission denied", the two most common errors in a trace.
• lsof: lists open files and sockets per process, or per file.
• eBPF: verified programs that run safely inside the kernel to trace events system-wide.
• bcc tools and bpftrace: ready-made eBPF tools, and a language for writing your own.

Pop quiz

Five questions: three on the ideas in this module, and two scenarios where you apply them. The order changes every time you take it, and 4 out of 5 passes.