A Troubleshooting Method: From Symptom to Root Cause

In this module, you'll learn to:

• Work an incident in a fixed order: symptom, change, scope, evidence, hypothesis, fix
• Check CPU, memory, disk, and network for utilization, saturation, and errors
• Run a 60-second checklist and read what each command is telling you

Before you start: finish Part 2, especially Processes, Signals & File Descriptors and Memory, the Page Cache & the OOM Killer. Open a shell on your lab VM with multipass shell lab, and install the tools:

sudo apt install -y sysstat stress-ng

Method beats memory

New engineers often troubleshoot by recognition: "last time it was the disk, so check the disk". That works for problems you've seen before and fails on everything else. Experienced engineers follow a method, so they make progress even on problems they've never met, and they can hand the investigation to someone else halfway through.

Three habits matter most:

• Define the symptom with a number. "Slow" isn't a symptom; "the checkout page takes 8 seconds instead of 300 ms since 14:05" is.
• Ask what changed. Most incidents follow a change. Check less /var/log/apt/history.log, journalctl --since "2 hours ago", last for recent sign-ins, and your deploy history.
• Write as you go. Note each command, what it showed, and the time. It stops you repeating checks, and it becomes the incident timeline.

Check every resource the same way

The USE method, from performance engineer Brendan Gregg, gives you a checklist that doesn't depend on guessing. For every resource, ask three questions:

• Utilization: how busy is it, as a percentage of its capacity?
• Saturation: is work queueing because it's too busy?
• Errors: is it failing?

Saturation is the one people miss. A disk at 60% utilization can still be the bottleneck if requests are queueing behind each other.

The first 60 seconds

Netflix's performance team published a checklist of ten commands to run in the first minute on any slow Linux server. It covers the USE questions for every resource:

Press Ctrl+C to stop each one after a few seconds. You won't use every column every time, but running the whole list stops you fixing the first thing you notice and missing the real cause.

Try it · A CPU-bound server. Load both CPUs, then run the checklist and see how it shows up:

stress-ng --cpu 2 --timeout 60s &
uptime                # the 1-minute load climbs towards 2
vmstat 1 5            # r is about 2, us is near 100, id near 0
mpstat -P ALL 1 3     # both CPUs busy
pidstat 1 3           # stress-ng is the process using it

High load, idle CPUs

Users say the server is sluggish. The load average is high, but when you look at top, the CPUs are mostly idle. Start the background job that's behind it, the way the scheduler starts it every night:

sudo systemd-run --unit=nightly-export \
  stress-ng --hdd 4 --hdd-opts dsync --timeout 600s

Now forget what you just ran. Your job: find what's slowing the server down, using the checklist rather than memory, and stop it. Try it before opening a hint.

Key terms

• Symptom: a precise, measured description of what's wrong, for whom, and since when.
• USE method: for every resource, check utilization, saturation, and errors.
• Saturation: work queueing because a resource is fully busy.
• Load average: the average number of processes running, waiting for CPU, or in D state.
• iowait (wa): CPU time spent idle while waiting for I/O.
• Steal (st): CPU time a virtual machine wanted but the hypervisor gave to another.
• Mitigate versus fix: restore service now, remove the cause after.

Pop quiz

Five questions: three on the ideas in this module, and two scenarios where you apply them. The order changes every time you take it, and 4 out of 5 passes.