Packages, Repositories & Where Software Lives

In this module, you'll learn to:

• Install, inspect, and remove software with apt, and translate those commands to dnf
• Add a signed third-party repository, and pin or hold a version
• Work out where a program came from when two copies of it exist

Before you start: finish Shell Survival. Open a shell on your lab VM with multipass shell lab.

Why software comes in packages

You could download a program and copy it into place. Then you'd also have to track its dependencies, check it wasn't tampered with, update it when a security fix comes out, and remember every file it put on the disk so you could remove it later. A package manager does all of that.

A package is an archive of files, plus metadata: its name, its version, what other packages it depends on, and scripts to run on install and removal. Packages come from repositories, servers that publish them along with a signed index. Ubuntu and Debian use .deb packages, managed with apt and dpkg. Red Hat, Rocky Linux, and Amazon Linux use .rpm packages, managed with dnf.

apt commands

sudo apt update              # refresh the package index. Installs nothing.
apt list --upgradable        # what has a newer version available?
sudo apt upgrade             # install those newer versions
sudo apt install -y tree     # install a package and its dependencies
apt show tree                # description, version, and dependencies
apt search "json processor"  # search names and descriptions
sudo apt remove tree         # remove the program, keep its config files
sudo apt purge tree          # remove it and its config files
sudo apt autoremove          # remove dependencies nothing needs any more

apt update and apt upgrade are different commands. update only refreshes the list of what's available; upgrade actually installs newer versions. Running upgrade without a recent update installs nothing new.

Which package, which file, which version

These questions come up constantly when you're working out what's on a server you've inherited:

dpkg -S /usr/bin/ssh          # which package installed this file?
dpkg -L openssh-client        # which files did this package install?
apt list --installed | wc -l  # how many packages are installed?
apt policy openssh-server     # installed version, newest available, and
                              #   which repository each version comes from
less /var/log/apt/history.log # what was installed or upgraded, and when

If you also work on Red Hat family systems, most of this translates directly:

Add a signed third-party repository

Ubuntu's own repositories don't carry every tool, or always the newest version. Vendors publish their own repositories. Add HashiCorp's and install Terraform, the tool behind the Terraform for Enterprise Production track on this site:

# 1. Download the vendor's signing key and store it as a keyring file
wget -O - https://apt.releases.hashicorp.com/gpg \
  | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg

# 2. Add the repository, trusting ONLY that key for it
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" \
  | sudo tee /etc/apt/sources.list.d/hashicorp.list

# 3. Refresh the index and install
sudo apt update
sudo apt install -y terraform

terraform version
apt policy terraform          # shows the version came from HashiCorp's repo

signed-by ties the repository to one key. Without it, a key added for one vendor could be used to sign packages that claim to come from another repository.

Hold a version when an upgrade must wait, for example until a team has tested the new release:

sudo apt-mark hold terraform     # apt upgrade will skip it
apt-mark showhold
sudo apt-mark unhold terraform

Where software lives, and which copy runs

When you type a command, the shell runs the first match it finds in the directories listed in $PATH, and on Ubuntu /usr/local/bin comes before /usr/bin. So a copy someone installed by hand silently wins over the packaged one, however often apt upgrades the packaged copy.

echo $PATH                 # the directories, in search order
type -a terraform          # EVERY copy on the PATH, the one that runs first

Keeping servers patched

Ubuntu servers install security updates automatically with unattended-upgrades, which is on by default. Some updates, such as a new kernel, only take effect after a reboot. When one is waiting, the file /var/run/reboot-required exists, and /var/run/reboot-required.pkgs lists which packages need it.

cat /var/run/reboot-required 2>/dev/null || echo "no reboot needed"
less /var/log/unattended-upgrades/unattended-upgrades.log

In production, reboot a few servers at a time, after checking that the rest can carry the load.

Every apt command fails

A teammate followed the newer install instructions on a vendor's website for a tool that was already installed. Since then, every apt command on the server fails, including the automatic security updates. Reproduce it after the lab above:

echo "deb [signed-by=/etc/apt/keyrings/hashicorp.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" \
  | sudo tee /etc/apt/sources.list.d/hashicorp-new.list

sudo apt update
sudo apt install -y tree

Your job: get apt working again without losing the HashiCorp repository. Try it before opening a hint.

Key terms

• Package: an archive of files, plus metadata, dependencies, and install scripts.
• Repository: a server that publishes packages with a signed index.
• apt update and apt upgrade: refresh the index, and install newer versions.
• signed-by: ties a repository to the one key allowed to sign it.
• Hold: stops a package from being upgraded until you release it.
• $PATH: the directories the shell searches for programs, in order. The first match runs.

Pop quiz

Five questions: three on the ideas in this module, and two scenarios where you apply them. The order changes every time you take it, and 4 out of 5 passes.