Incident Review: "trust" Authentication Failed, Through a Unix Socket

About this review. This is a real incident from a production platform running on RHEL 8 servers in AWS, written up from the engineering team's notes. Every server name, path, account, product name, and credential is replaced with a generic one or removed; the errors, commands, and fixes are otherwise as recorded.

A database that moved

The batch job (etl.service) runs under systemd as the service account appsvc. Each cycle ends with a maintenance phase that runs SQL by shelling out to the psql command.

Its database used to live on the same server. It had moved to Amazon RDS, reached through a local PgBouncer connection pooler on port 6432. The pooler's authentication was configured like this (credentials removed):

listen_port     = 6432
unix_socket_dir = /var/run/postgresql
auth_type       = trust
auth_file       = /opt/app/config/pg_auth

With auth_type = trust, PgBouncer doesn't check a password, but the user name must still be listed in auth_file. That file listed only the database user postgres.

A precise error, easy to misread

psql: connection to server on socket "/run/postgresql/.s.PGSQL.6432" failed:
ERROR: "trust" authentication failed

The team first checked the job's database configuration files. Both had the correct host, port, user, and password, which made the error look impossible.

Reproduce it as the service account

The breakthrough was running psql exactly the way the job does, as the same Linux account, with the same (missing) options:

# As the job runs it: no host, no user. Fails.
sudo -u appsvc psql -d appdb -p 6432 -c "select 1;"

# With an explicit TCP host and database user. Works.
sudo -u appsvc psql -h 127.0.0.1 -U postgres -d appdb -p 6432 -c "select 1;"

# Which user and server does each one really reach?
sudo -u appsvc psql -h 127.0.0.1 -U postgres -d appdb -p 6432 \
  -c "select current_user, inet_server_addr();"

Then they read how the maintenance code actually calls psql:

grep -n "psql" /path/to/vendor/sweep.rb
grep -n -E "Open3|system\(|%x\[" /path/to/vendor/sweep.rb

Every call was built as a raw shell command along the lines of psql -d <database> -p <port> ..., with no host and no user. That code path never read the configuration files, which is why their correct values didn't help.

Two silent defaults

When psql isn't told otherwise, it fills in two things for you:

PgBouncer received a connection for user appsvc, found no such user in its auth_file, and rejected it: "trust" authentication failed.

When the database was local, these defaults had happened to work. The move didn't break the job; as the team's notes put it, it exposed a design flaw that had always been there.

Two tested approaches

Approach 1 · Supply the connection settings from systemd. psql reads standard environment variables when options are missing. Setting them on the service, in a drop-in, makes every psql call the job spawns use TCP and the right user, without touching the vendor's code:

sudo systemctl edit etl
#   [Service]
#   Environment="PGHOST=127.0.0.1"
#   Environment="PGPORT=6432"
#   Environment="PGDATABASE=appdb"
#   Environment="PGUSER=postgres"
#   (plus the password; see the note below)
sudo systemctl daemon-reload
sudo systemctl restart etl
systemctl show etl -p Environment

Approach 2 · Patch the calls. After backing up the vendor's file, every psql call in it, including two built with Open3.popen3, was changed to pass -U with the configured database user explicitly.

Both were tested and worked. The notes recommend approach 1 as the cleaner, permanent fix: it changes no vendor code, survives package upgrades, and covers every current and future psql call the job makes.

A note on the password. Values in a unit's Environment= lines can be read by any user on the server with systemctl show, and systemd's documentation advises against putting secrets there. Safer places for a database password are a ~/.pgpass file owned by the service account with mode 600, which psql reads automatically, or an EnvironmentFile= readable only by root.

Lessons

• Never rely on client defaults in automation. Pass host, port, user, and database explicitly, or set them once for the whole service.
• Reproduce as the service account, with the exact command the application builds. The same psql worked or failed depending only on its options and user.
• A migration is a test of hidden assumptions. After moving a dependency, check how each consumer connects, not only that the new endpoint answers.
• Keep credentials out of commands and unit files. Command lines and Environment= values are visible to other users; use protected files instead.

Where this track teaches the skills

• Users, sudo & SSH: sudo -u to run a command as another account.
• systemd: Services, Timers & Logs: drop-ins and Environment= for a service.
• Networking Basics: TCP connections versus local sockets, and who can reach what.
• Automation: Safe Bash, cloud-init & Ansible: why scheduled and service jobs don't share your terminal's environment.

Pop quiz

Five questions: three on the ideas in this review, and two scenarios where you apply them. The order changes every time you take it, and 4 out of 5 passes.