systemd: Services, Timers & Logs

In this module, you'll learn to:

• Start, stop, enable, and inspect services with systemctl, and read their logs with journalctl
• Write a unit file with a restart policy and basic hardening, and change it with drop-in overrides
• Schedule work with a systemd timer, and diagnose a service that won't stay up

Before you start: read The Boot Process. Open a shell on your lab VM with multipass shell lab.

systemd keeps services running

systemd is PID 1, the first process the kernel starts. Besides booting the system, it supervises everything long-running: web servers, databases, agents, your own apps. You describe each one in a unit file, and systemd starts it in the right order, restarts it when it crashes, captures its output, and limits what it can touch.

systemctl and journalctl

systemctl status ssh            # running? since when? main PID, last log lines
sudo systemctl restart ssh      # stop, then start
sudo systemctl reload ssh       # re-read config without stopping (if supported)
sudo systemctl enable ssh       # start at boot
sudo systemctl disable ssh      # don't start at boot
systemctl is-enabled ssh
systemctl list-units --type=service --state=running
systemctl --failed              # anything that failed: check this on any sick server

systemctl cat ssh               # the unit file, plus any overrides
journalctl -u ssh               # this service's logs
journalctl -u ssh -f            # follow live, like tail -f
journalctl -u ssh --since "1 hour ago"
journalctl -p err -b            # errors from every service since boot

enable and start are separate. start runs a service now, and enable makes it start at every boot. sudo systemctl enable --now name does both, and forgetting enable is the classic reason a service is missing after a reboot.

Unit files shipped by packages live in /usr/lib/systemd/system/. Your own units and changes go in /etc/systemd/system/, which takes priority. Never edit the packaged files: the next package update overwrites them.

Turn a script into a service

Create a small web app and a dedicated system user to run it:

sudo useradd --system --no-create-home --shell /usr/sbin/nologin hello
sudo mkdir -p /opt/hello

sudo tee /opt/hello/app.py <<'EOF'
import http.server, os
port = int(os.environ.get("PORT", "8080"))
class Handler(http.server.BaseHTTPRequestHandler):
    def do_GET(self):
        self.send_response(200)
        self.end_headers()
        self.wfile.write(b"hello from systemd\n")
print(f"listening on {port}", flush=True)
http.server.HTTPServer(("0.0.0.0", port), Handler).serve_forever()
EOF

Now the unit file:

sudo tee /etc/systemd/system/hello.service <<'EOF'
[Unit]
Description=Hello demo web app
After=network-online.target
Wants=network-online.target

[Service]
User=hello
Environment=PORT=8080
ExecStart=/usr/bin/python3 /opt/hello/app.py
Restart=on-failure
RestartSec=2
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now hello
systemctl status hello
curl localhost:8080
journalctl -u hello -n 5

Each line earns its place:

Test the restart policy by killing the process the way a crash would:

sudo systemctl kill -s KILL hello
sleep 3
systemctl status hello       # a new main PID, and "activating" or "running" again

Drop-in overrides

To change a unit, whether it's yours or one from a package, don't edit the original file. Add a drop-in: a small file that overrides individual settings and survives package updates.

sudo systemctl edit hello

This opens an editor on /etc/systemd/system/hello.service.d/override.conf. Add these lines, then save:

[Service]
Environment=PORT=9090

sudo systemctl restart hello     # `systemctl edit` already reloaded systemd
curl localhost:9090
systemctl cat hello              # the original file, then the override

systemctl cat is how you find out why a service doesn't behave the way its main file says. Someone may have added a drop-in.

Timers instead of cron

cron still works, but systemd timers log every run to the journal, can catch up on runs missed while the server was off, and show when they'll fire next. A timer starts a service of the same name:

sudo tee /etc/systemd/system/disk-report.service <<'EOF'
[Unit]
Description=Log disk usage

[Service]
Type=oneshot
ExecStart=/usr/bin/df -h /
EOF

sudo tee /etc/systemd/system/disk-report.timer <<'EOF'
[Unit]
Description=Log disk usage every 15 minutes

[Timer]
OnCalendar=*:0/15
Persistent=true

[Install]
WantedBy=timers.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now disk-report.timer
systemctl list-timers disk-report.timer      # when it fires next
sudo systemctl start disk-report.service     # run it once now
journalctl -u disk-report -n 5

OnCalendar=*:0/15 means "every hour, at minute 0 and every 15 minutes after". Persistent=true runs a missed job as soon as the server is back up.

The service that won't stay up

A teammate tightened security on the hello service last night. This morning it's down, and the monitoring shows it restarted several times before giving up. Reproduce the change:

sudo mkdir -p /etc/systemd/system/hello.service.d
sudo tee /etc/systemd/system/hello.service.d/10-security.conf <<'EOF'
[Service]
User=helo
EOF
sudo systemctl daemon-reload
sudo systemctl restart hello

Your job: find out why the service won't start, fix it, and get it running again. Try it before opening a hint.

Key terms

• Unit: anything systemd manages: a service, timer, mount, or target.
• Unit file: the configuration for a unit. Yours go in /etc/systemd/system/.
• daemon-reload: makes systemd re-read unit files after you change them.
• Drop-in override: a small file in name.service.d/ that changes individual settings.
• Restart policy: when systemd restarts a crashed service, and how often before it gives up.
• Timer: a unit that starts a service on a schedule, replacing cron.
• journald and journalctl: where service output is stored, and the tool that reads it.

Pop quiz

Five questions: three on the ideas in this module, and two scenarios where you apply them. The order changes every time you take it, and 4 out of 5 passes.