Storage, Filesystems & LVM

In this module, you'll learn to:

• Map disks, partitions, filesystems, and mount points with lsblk, df, and findmnt
• Find what's using disk space, and recognise when inodes run out instead
• Create an LVM volume, grow it without unmounting, and recover space held by deleted files

Before you start: finish Files, Inodes & Permissions and The Boot Process. Open a shell on your lab VM with multipass shell lab.

From a disk to a directory

Storage on Linux is built in layers. Knowing which layer you're looking at is half of every storage problem.

lsblk                  # disks and partitions, and where each is mounted
lsblk -f               # plus filesystem type, label, and UUID
findmnt                # every mount, as a tree
df -hT                 # space used and free per filesystem, with its type

ext4 is the default on Ubuntu and Debian. xfs is the default on RHEL family systems and handles very large files well. One practical difference: ext4 can be shrunk, xfs can only grow.

What's filling the disk

df answers how full is each filesystem. du answers which directories use the space. Together they find almost every full disk:

df -h                                          # which filesystem is full?
sudo du -xh --max-depth=1 / 2>/dev/null | sort -h | tail -8
                                               # biggest directories on that filesystem
sudo du -xh --max-depth=1 /var 2>/dev/null | sort -h | tail -8
                                               # then go one level deeper, and repeat

-x keeps du on one filesystem, so it doesn't wander into other mounts. The usual culprits are logs in /var/log, container images in /var/lib/docker or /var/lib/containerd, old kernels, and forgotten backups.

A filesystem can also run out of inodes while plenty of space remains, as Files, Inodes & Permissions explained. Every new file then fails with "No space left on device", even though df -h shows free space:

df -i                  # inode use per filesystem; look for 100% IUse

Temporary and permanent

mount attaches a filesystem now; /etc/fstab attaches it at every boot.

# device                                    mount point  type  options           dump pass
UUID=0a1b2c3d-1111-2222-3333-444455556666   /data        ext4  defaults,nofail   0    2

After any change: sudo findmnt --verify and sudo mount -a, before you reboot.

A mount that stopped answering

This one happened while this track was being written, on a workstation rather than a server, but the lesson is the same one production teams meet with NFS and EFS.

A Linux terminal running under WSL (Linux on Windows) refused to open a project folder on the D: drive. It failed with chdir(/mnt/d/...) failed 5 and dropped into / instead. Every path under /mnt/d returned Input/output error, even ls /mnt/d itself. Meanwhile Windows opened the same folder normally, and its permissions looked normal. So the disk and the files were fine. What had broken was the mount: WSL reaches Windows drives through a network-style filesystem mounted at /mnt/d, and that connection had failed.

# Is the failing path inside a mount, and what is it?
findmnt -T /mnt/d/project      # the mount that contains this path, its source and options
ls /mnt/d                      # Input/output error: the mount is broken, not the files

# Copy the options from a healthy mount of the same kind
findmnt -o TARGET,SOURCE,FSTYPE,OPTIONS /mnt/c

# Recover: detach the broken mount now, then mount it again
sudo umount -l /mnt/d
sudo mount -t drvfs D: /mnt/d -o uid=1000,gid=1000

umount -l is a lazy unmount: it detaches the mount from the directory tree immediately, and finishes cleaning up once no process is still using it. That's what lets you recover a mount that's too broken to unmount normally.

The first remount used different options, and every file came back owned by root instead of the user. Git refuses to work in a repository owned by another user ("dubious ownership"), so that would have looked like a brand-new problem. Copying the options from the healthy C: drive mount (uid=1000,gid=1000) fixed it. For mounts listed in /etc/fstab, sudo mount /path remounts with the options recorded there, which avoids this mistake entirely.

On servers, the same pattern shows up with network filesystems. An NFS or EFS mount can go stale after a network interruption or a change on the file server: commands fail with Stale file handle, or processes touching the mount hang in D state, as in Processes, Signals & File Descriptors. The approach doesn't change:

• When every path under one directory fails and the rest of the system is fine, suspect the mount, not the files. findmnt -T <path> shows which mount you're actually in.
• Check the other side first: is the file server reachable, and is the export still there?
• Remount with the original options, ideally from /etc/fstab, and check ownership afterwards. A remount that "works" with the wrong options causes the next problem.

Volumes you can grow while they're in use

LVM (Logical Volume Manager) puts a flexible layer between disks and filesystems. Disks become physical volumes (PVs), PVs are pooled into a volume group (VG), and you carve logical volumes (LVs) out of the pool. When a volume fills up, you add a disk to the pool and grow the volume, with the filesystem still mounted.

The lab uses loop devices, ordinary files that act as disks, so you can practise without attaching real ones:

sudo apt install -y lvm2
truncate -s 1G ~/disk1.img ~/disk2.img
LOOP1=$(sudo losetup -f --show ~/disk1.img)
LOOP2=$(sudo losetup -f --show ~/disk2.img)
echo "$LOOP1 $LOOP2"

# One disk into a pool, one volume using all of it, an ext4 filesystem on top
sudo pvcreate "$LOOP1"
sudo vgcreate datavg "$LOOP1"
sudo lvcreate -n datalv -l 100%FREE datavg
sudo mkfs.ext4 /dev/datavg/datalv
sudo mkdir -p /mnt/data
sudo mount /dev/datavg/datalv /mnt/data
df -h /mnt/data                 # about 1 GB

# The volume is filling up: add the second disk and grow, while mounted
sudo pvcreate "$LOOP2"
sudo vgextend datavg "$LOOP2"
sudo lvextend -r -l +100%FREE /dev/datavg/datalv
df -h /mnt/data                 # about 2 GB, with no unmount and no downtime

sudo pvs; sudo vgs; sudo lvs    # the three layers, summarised

-r on lvextend resizes the filesystem in the same step. Without it, the volume grows but the filesystem inside doesn't, and df shows no change. Cloud disks work the same way: after enlarging an EBS volume, for example, you still grow the partition and the filesystem inside the server.

Full disk, empty directory

Monitoring alerts that /mnt/data is about 80% full. Someone already deleted the big log file to free space, yet the alert hasn't cleared. Reproduce it on the volume from the lab:

sudo dd if=/dev/zero of=/mnt/data/app.log bs=1M count=1500
sudo systemd-run --unit=applog bash -c 'exec 3>>/mnt/data/app.log; sleep infinity'
sudo rm /mnt/data/app.log

df -h /mnt/data            # still about 80% used
sudo du -sh /mnt/data      # almost nothing

Your job: explain the difference between the two numbers and get the space back. Try it before opening a hint.

When you're done, remove the lab volume:

sudo umount /mnt/data
sudo lvremove -y datavg/datalv
sudo vgremove datavg
sudo pvremove "$LOOP1" "$LOOP2"
sudo losetup -d "$LOOP1" "$LOOP2"
rm ~/disk1.img ~/disk2.img

Key terms

• Block device: a disk, or something that acts like one, such as /dev/sda.
• Filesystem: the structure on a device that holds inodes, directories, and data. ext4 and xfs are the common ones.
• Mount point: the directory where a filesystem appears in the tree.
• df and du: space per filesystem, and space per directory.
• Inode exhaustion: no inodes left for new files, while space remains.
• LVM (PV, VG, LV): disks pooled into a group and carved into volumes you can grow.
• Lazy unmount (umount -l): detach a mount now, and finish cleaning up once nothing is using it; the way to recover a broken mount.
• Deleted-but-open file: a file with no name left, whose space stays used until the last process closes it.

Pop quiz

Five questions: three on the ideas in this module, and two scenarios where you apply them. The order changes every time you take it, and 4 out of 5 passes.