Incident Review: Two Faults Behind One Failed Daemon

About this review. This is a real incident from a production platform running on RHEL 8 servers in AWS, written up from the engineering team's notes. Every server name, path, account, and product name is replaced with a generic one; the causes, commands, and fix are otherwise as recorded.

A daemon with a hard dependency

The daemon (appd.service) is started by systemd. Its unit declares a hard dependency on the local connection pooler, PgBouncer, which sits between applications on the server and the managed PostgreSQL databases:

[Unit]
Description=Application back-end daemon
After=pgbouncer.service
Requires=pgbouncer.service

[Service]
Type=simple
ExecStartPre=/bin/bash /opt/app/systemd_scripts/appd_prestart.sh
ExecStart=/opt/app/appd/appd --monitor -l3

[Install]
WantedBy=multi-user.target

Requires= means the daemon is only useful when the pooler, and the databases behind it, are working. Its ExecStartPre= script runs set-up steps before the daemon itself starts.

The start-up step fails

The service failed in its ExecStartPre= step. The investigation followed the standard sequence from systemd: Services, Timers & Logs:

systemctl daemon-reload
systemctl status appd
systemctl cat appd.service                 # the unit, plus any drop-ins changing it
journalctl -u appd.service -n 50 --no-pager
tail -40 /opt/app/log/appd.log

The pooler pointed at another environment

The daemon depends on its databases through the pooler, so the team tested every database through the pooler, the same way the daemon reaches them:

for db in db_one db_two db_three; do
  echo "Testing $db..."
  psql -h localhost -p 6432 -U postgres -d "$db" -c "SELECT 1;" \
    && echo "$db OK" || echo "$db FAILED"
done

# Which backend does each pooler database name map to?
psql -h 127.0.0.1 -p 6432 -U postgres pgbouncer -c "show databases;"

show databases lists each name the pooler accepts and the real database host it forwards to. On this QA server, the mappings still pointed at the Stage environment's database cluster, so back-end database connections failed.

Non-root, and no shared memory

Separately, the daemon had been changed to run as the non-root application account instead of root. Running services without root is normally good practice. But this daemon creates shared memory segments when it starts, and as the non-root user that failed with errno 13, Permission denied, which failed the ExecStartPre= step.

Shared memory here means System V IPC: memory segments the kernel keeps for processes to share, each with an owner and permissions, much like files. ipcs -m lists the segments that exist, with their owners and permissions, which is the place to start when a process can't create or attach to one.

Both, then verify through the same path

1. Pooler: the database mappings in the pooler's configuration were changed to point at the QA database endpoint, and connectivity was tested again through port 6432.
2. Daemon: the run-as-user change was reverted, so the daemon runs as root as originally designed:

systemctl daemon-reload
systemctl restart appd
systemctl status appd

With both fixed, database connectivity and the daemon's start-up were restored.

Why two faults are harder than one

With two independent causes, fixing either one alone doesn't make the symptom go away. The natural conclusion, "that wasn't it", is wrong, and the correct fix gets reverted. Two habits avoid that:

• Verify each layer separately after each change: database connectivity through the pooler, then the daemon's own start-up, rather than only "does the service start?".
• Keep notes of what each change proved, so a change that fixed one layer isn't undone because the overall symptom remained.

Lessons

• Check environment wiring after building an environment. A copied configuration that still points at another environment is a classic, and show databases against the pooler catches it in seconds.
• Change a service's run-as user deliberately, and test it. Moving a daemon off root is worth doing, but it needs a test in a lower environment, and a check of everything the daemon creates at start-up: files, sockets, and shared memory.
• Read systemctl cat before trusting a unit file. A drop-in can change the user, the start-up steps, or the environment without touching the main file.

Where this track teaches the skills

• systemd: Services, Timers & Logs: Requires=, ExecStartPre=, drop-ins, and systemctl cat.
• Processes, Signals & File Descriptors: what a process owns and which user it runs as.
• A Troubleshooting Method: one hypothesis and one change at a time, with notes.
• Capstone: the same situation, several faults at once, in a lab.

Pop quiz

Five questions: three on the ideas in this review, and two scenarios where you apply them. The order changes every time you take it, and 4 out of 5 passes.