Automation: Safe Bash, cloud-init & Ansible

In this module, you'll learn to:

• Write Bash scripts that stop on errors, clean up after themselves, and can safely run twice
• Explain what cloud-init does at first boot, and where it fits next to images and Ansible
• Write and run an idempotent Ansible playbook, and diagnose jobs that only fail when scheduled

Before you start: finish Shell Survival and systemd: Services, Timers & Logs. Open a shell on your lab VM with multipass shell lab.

From commands to code

Typing the same commands on every server doesn't scale, and it drifts: one server gets a slightly different setting, and nobody remembers why. Automation turns those commands into code you can review, test, and run again. Three tools cover most of it, each at a different stage of a server's life:

• cloud-init runs once, at first boot: users, keys, packages, and a bootstrap command.
• Ansible keeps every server in the desired state, run after run.
• Bash is the glue for one-off tasks, health checks, and small jobs in between.

Scripts that fail safely

Shell Survival showed a script that printed "Backup complete" after failing. A safe script starts like this:

#!/usr/bin/env bash
set -euo pipefail                     # stop on errors, unset variables, failed pipes

readonly BACKUP_DIR="/var/backups/app"
tmp="$(mktemp -d)"                    # a private temporary directory
trap 'rm -rf "$tmp"' EXIT             # always removed, even if the script fails

log() { echo "$(date -Is) $*" >&2; }  # messages go to stderr, with a timestamp

main() {
  [[ -d "$BACKUP_DIR" ]] || { log "missing $BACKUP_DIR"; exit 1; }
  log "archiving to $tmp"
  tar -czf "$tmp/app.tgz" -C "$BACKUP_DIR" .
  mv "$tmp/app.tgz" "/srv/backups/app-$(date +%F).tgz"
  log "done"
}

main "$@"

The rules that matter most:

• set -euo pipefail at the top of every script.
• Quote every variable: "$file", not $file. Without quotes, a name with a space becomes two arguments, and an empty variable can turn rm -rf "$dir/" into something very different.
• trap ... EXIT for clean-up, so temporary files and locks don't outlive a failure.
• Check with ShellCheck: sudo apt install -y shellcheck, then shellcheck script.sh catches most quoting mistakes and common bugs before they run.

Safe to run twice

A script is idempotent if running it a second time changes nothing and breaks nothing. That's what lets you rerun automation after a failure, or on a schedule, without fear:

Configure at first boot

Cloud images run cloud-init on first boot. It reads user data, supplied when the server is created, and applies it: users, SSH keys, packages, files, and commands. You've already used it: the keys.yaml in Users, sudo & SSH was cloud-init user data. A fuller example:

#cloud-config
package_update: true
packages: [nginx, prometheus-node-exporter]
write_files:
  - path: /var/www/html/index.html
    content: "Built by cloud-init\n"
runcmd:
  - systemctl enable --now nginx

# On your laptop: a new VM from that file
multipass launch 24.04 --name web --cloud-init web.yaml

# On the VM: did it work, and what did it do?
cloud-init status --long
sudo less /var/log/cloud-init-output.log

Keep user data small: enough to make the server reachable and hand over to configuration management. Big bootstrap scripts are hard to test and only run once.

Describe the state, not the steps

Ansible connects to servers over SSH and applies a playbook: a list of tasks that each describe a desired state ("nginx is installed", "this file has this content", "the service is running"). Each task checks the current state first and changes only what's different, so playbooks are idempotent by design.

On the lab VM, install Ansible and write a playbook that configures the VM itself:

sudo apt install -y ansible
mkdir -p ~/ansible && cd ~/ansible

# site.yml
- hosts: all
  become: true
  tasks:
    - name: Install nginx
      ansible.builtin.apt:
        name: nginx
        state: present
        update_cache: true

    - name: Home page
      ansible.builtin.copy:
        dest: /var/www/html/index.html
        content: "Managed by Ansible\n"

    - name: nginx is running and starts at boot
      ansible.builtin.service:
        name: nginx
        state: started
        enabled: true

ansible-playbook -i localhost, -c local site.yml    # first run: changed=2 or 3
ansible-playbook -i localhost, -c local site.yml    # second run: changed=0
curl localhost

The second run reporting changed=0 is idempotency in action. In real use, the inventory (-i) lists your servers, and Ansible reaches them over SSH, through a bastion with the ProxyJump setup from Users, sudo & SSH.

Works by hand, fails in cron

A teammate's nightly report script works perfectly when they run it. Scheduled in cron, it has never produced a single report. Set it up exactly as they did:

mkdir -p ~/bin ~/reports
cat > ~/bin/stamp <<'EOF'
#!/bin/bash
date +%F-%T
EOF
cat > ~/bin/nightly-report.sh <<'EOF'
#!/bin/bash
set -euo pipefail
stamp > "$REPORT_DIR/last-run.txt"
EOF
chmod +x ~/bin/stamp ~/bin/nightly-report.sh

echo 'export PATH="$HOME/bin:$PATH"' >> ~/.bashrc
echo 'export REPORT_DIR="$HOME/reports"' >> ~/.bashrc
source ~/.bashrc

nightly-report.sh && cat ~/reports/last-run.txt          # works by hand

# Schedule it every minute, for the lab
( crontab -l 2>/dev/null; echo "* * * * * $HOME/bin/nightly-report.sh" ) | crontab -

Wait two minutes, then check ~/reports/last-run.txt. It never changes. Your job: find out why, and make the scheduled run work. Try it before opening a hint.

Key terms

• set -euo pipefail: stop on errors, unset variables, and failed pipelines.
• trap ... EXIT: clean-up that runs however the script ends.
• Idempotent: running it again changes nothing and breaks nothing.
• cloud-init and user data: first-boot configuration, and the instructions it applies.
• Ansible playbook: a list of tasks, each describing a desired state.
• Inventory: the list of servers Ansible manages.
• cron's environment: a minimal PATH and no shell startup files, unlike your terminal.

Pop quiz

Five questions: three on the ideas in this module, and two scenarios where you apply them. The order changes every time you take it, and 4 out of 5 passes.