Capstone: An Incident with Three Unknown Faults

multipass launch 24.04 --name capstone --cpus 2 --memory 2G --disk 10G
multipass shell capstone

sudo apt install -y nginx postgresql python3-psycopg2

sudo -u postgres psql -c "CREATE ROLE app LOGIN PASSWORD 'lab-only-password';"
sudo -u postgres psql -c "CREATE DATABASE shop OWNER app;"
PGPASSWORD=lab-only-password psql -h 127.0.0.1 -U app -d shop -c \
  "CREATE TABLE orders (id serial PRIMARY KEY, item text);
   INSERT INTO orders (item) VALUES ('keyboard'), ('monitor'), ('cable');"

sudo useradd --system --no-create-home --shell /usr/sbin/nologin shop
sudo mkdir -p /opt/shop
sudo tee /opt/shop/app.py <<'EOF'
import http.server, os
import psycopg2

DB = dict(host=os.environ.get("DB_HOST", "127.0.0.1"), dbname="shop", user="app",
          password=os.environ["DB_PASSWORD"], connect_timeout=3)

class Handler(http.server.BaseHTTPRequestHandler):
    def do_GET(self):
        try:
            conn = psycopg2.connect(**DB)
            try:
                cur = conn.cursor()
                cur.execute("SELECT count(*) FROM orders")
                body, status = f"orders: {cur.fetchone()[0]}\n".encode(), 200
            finally:
                conn.close()
        except Exception as e:
            print(f"database error: {e}", flush=True)
            body, status = b"database unavailable\n", 500
        self.send_response(status)
        self.end_headers()
        self.wfile.write(body)

http.server.ThreadingHTTPServer(("127.0.0.1", 5000), Handler).serve_forever()
EOF

sudo tee /etc/systemd/system/shop.service <<'EOF'
[Unit]
Description=Shop app
After=network-online.target postgresql.service
Wants=network-online.target

[Service]
User=shop
Environment=DB_PASSWORD=lab-only-password
ExecStart=/usr/bin/python3 /opt/shop/app.py
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now shop
curl -s localhost:5000          # orders: 3

sudo tee /etc/nginx/sites-available/shop <<'EOF'
server {
    listen 80 default_server;
    location / {
        proxy_pass http://127.0.0.1:5000;
        proxy_connect_timeout 3s;
        proxy_read_timeout 10s;
    }
}
EOF
sudo rm -f /etc/nginx/sites-enabled/default
sudo ln -s /etc/nginx/sites-available/shop /etc/nginx/sites-enabled/shop
sudo nginx -t && sudo systemctl reload nginx

curl -s localhost/              # orders: 3, through nginx
ip -br addr                     # note the VM's address

cat > ~/break.sh <<'EOF'
#!/bin/bash
# Capstone fault injector. Run it, but don't read it until you've finished.
set -euo pipefail
f1() { sed -i 's|127.0.0.1:5000|127.0.0.1:5001|' /etc/nginx/sites-available/shop; systemctl reload nginx; }
f2() { mkdir -p /etc/systemd/system/shop.service.d
       printf '[Service]\nEnvironment=DB_HOST=127.0.0.2\n' > /etc/systemd/system/shop.service.d/50-db.conf
       systemctl daemon-reload; systemctl restart shop; }
f3() { sudo -u postgres psql -qc "ALTER ROLE app CONNECTION LIMIT 0;"; }
f4() { systemctl stop nginx; systemd-run --quiet --unit=legacy-health nc -lk 80; }
f5() { nft add table inet capstone
       nft add chain inet capstone input '{ type filter hook input priority 0; policy accept; }'
       nft add rule inet capstone input iifname != "lo" tcp dport 80 drop; }
f6() { systemctl disable --now postgresql >/dev/null 2>&1; }
for f in $(printf '%s\n' f1 f2 f3 f4 f5 f6 | shuf -n 3 | sort); do "$f"; done
echo "Three faults injected. Users report the shop is down. Start your timeline: $(date +%T)"
EOF

sudo bash ~/break.sh

Good action items are specific and checkable: "add an external HTTP check that alerts within 2 minutes", not "be more careful".

### Implementation notes

- **Restore service first, then dig.** If a rollback or restart brings customers back, do it, capture evidence first if it takes seconds, and find the root cause afterwards.
- **One person changes things, everyone else watches.** In a real incident with several people, uncoordinated changes create new faults faster than you can find the old ones.
- **Communicate on a schedule.** A short update every 15 to 30 minutes ("still investigating; the database is ruled out") stops people interrupting the person fixing it.
- **Test from where customers are.** One of the six faults looks completely fine from the server itself, and in real incidents that kind of fault is common.