Containers from Scratch: Namespaces, cgroups & Overlay Filesystems

In this module, you'll learn to:

• Explain the Linux features a container is made of: namespaces, cgroups, an overlay filesystem, capabilities, and seccomp
• Build a working container by hand with unshare, chroot, and a cgroup
• Find those same pieces in a running Docker container, and diagnose container networking from the host

Before you start: finish Part 2, especially Processes, Memory, and Networking Basics. Open a shell on your lab VM with multipass shell lab.

A container is a process with blinkers on

On the host, a container's main process is just another process: you can see it in ps, with a normal PID. What makes it a container is how the kernel restricts it:

There's no separate kernel, as there would be in a VM. Every container shares the host's kernel, which is why containers start in milliseconds and why a kernel vulnerability matters to all of them.

Build the isolation by hand

First, a tiny root filesystem. busybox is a single static program that acts as sh, ls, ps, and hundreds of other commands:

sudo apt install -y busybox-static
mkdir -p ~/ctr/rootfs/{bin,proc,sys,tmp}
cp /bin/busybox ~/ctr/rootfs/bin/
sudo chroot ~/ctr/rootfs /bin/busybox --install -s /bin
ls ~/ctr/rootfs/bin | head

Now start a shell in new namespaces, with that directory as its root:

sudo unshare --uts --pid --mount --net --ipc --fork \
  chroot ~/ctr/rootfs /bin/sh

Inside it:

mount -t proc proc /proc
hostname container-1
hostname           # container-1, while the host keeps its own name
ps                 # PID 1 is this shell; the host's processes are invisible
ip link            # only a loopback interface: a network of its own, empty
exit

Container runtimes also use user namespaces (root inside maps to an ordinary user outside) and cgroup namespaces. lsns lists every namespace on the host.

Add a memory limit

cgroups (version 2 on current distributions) are managed through files under /sys/fs/cgroup. Create one, limit it to 50 MB, and start the container inside it:

sudo mkdir /sys/fs/cgroup/demo
echo 50M | sudo tee /sys/fs/cgroup/demo/memory.max
echo 0   | sudo tee /sys/fs/cgroup/demo/memory.swap.max

# The inner shell moves itself into the cgroup, then becomes the container
sudo sh -c 'echo $$ > /sys/fs/cgroup/demo/cgroup.procs &&
  exec unshare --uts --pid --mount --net --ipc --fork chroot '"$HOME"'/ctr/rootfs /bin/sh'

Inside the container, try to hold 100 MB in a shell variable:

x=$(head -c 100000000 /dev/zero | tr '\0' a); echo "survived"

You never see "survived": the kernel's OOM killer ends it at the 50 MB limit, exactly as in Memory, the Page Cache & the OOM Killer. Back on the host:

grep oom_kill /sys/fs/cgroup/demo/memory.events    # the kill, counted
sudo rmdir /sys/fs/cgroup/demo                     # remove the empty cgroup

Image layers and copy-on-write

Container images are read-only layers. overlayfs stacks them and puts a writable layer on top. Changing a file copies it up into the writable layer, so the image itself never changes, and a hundred containers can share one copy of it.

mkdir -p ~/ovl/{lower,upper,work,merged}
echo "from the image" > ~/ovl/lower/a.txt
sudo mount -t overlay overlay \
  -o lowerdir=$HOME/ovl/lower,upperdir=$HOME/ovl/upper,workdir=$HOME/ovl/work \
  $HOME/ovl/merged

cat ~/ovl/merged/a.txt                        # from the image
echo "changed in the container" | sudo tee ~/ovl/merged/a.txt
cat ~/ovl/lower/a.txt                         # unchanged: still "from the image"
cat ~/ovl/upper/a.txt                         # the copied-up, changed version
sudo umount ~/ovl/merged

Deleting the writable layer resets the container to the image. That's why anything written inside a container is lost when it's removed, unless it's on a mounted volume.

Find the same pieces in Docker

sudo apt install -y docker.io
sudo docker run -d --name web --memory 64m nginx:alpine

PID=$(sudo docker inspect -f '{{.State.Pid}}' web)
ps -o pid,user,cmd -p "$PID"         # nginx, as an ordinary host process
sudo lsns -p "$PID"                  # the namespaces it lives in
cat /proc/"$PID"/cgroup              # its cgroup...
cat /sys/fs/cgroup/system.slice/docker-$(sudo docker inspect -f '{{.Id}}' web).scope/memory.max
                                     # ...and the 64 MB limit: 67108864 bytes
findmnt -t overlay | head -3         # the overlay root filesystems
grep Cap /proc/"$PID"/status         # its reduced set of capabilities
sudo docker rm -f web

--privileged turns off nearly all of these protections, giving the container every capability and the host's devices. Treat it as "root on the host", and avoid it outside special cases. Kubernetes runs containers with exactly these features, and the Kubernetes track builds on everything in this module.

Containers lost the internet

A security hardening script ran on the Docker host this morning. The host itself is fine, but container builds now fail with "bad address" when they download packages. Reproduce it:

sudo docker run --rm alpine ping -c 2 1.1.1.1      # works, before the change
sudo sysctl -w net.ipv4.ip_forward=0                # what the script changed
sudo docker run --rm alpine wget -qO- -T 5 http://example.com

Your job: find out why containers can't get out when the host can, and fix it so it stays fixed. Try it before opening a hint.

Key terms

• Namespace: a kernel feature that gives a process its own view of one resource, such as PIDs or networks.
• cgroup: a group of processes the kernel limits and measures together.
• overlayfs: stacks read-only layers with a writable layer on top, copying files up on change.
• Capabilities: root's powers split into pieces, so a container gets only the few it needs.
• seccomp: a filter that blocks risky system calls.
• --privileged: turns off the container's protections; effectively root on the host.
• ip_forward: the setting that lets the host route packets for containers.

Pop quiz

Five questions: three on the ideas in this module, and two scenarios where you apply them. The order changes every time you take it, and 4 out of 5 passes.