Incident Review: SSH Worked by Hand but Failed for the Service

About this review. This is a real incident from a production platform running on RHEL 8 servers in AWS, written up from the engineering team's notes. Every server name, path, account, and product name is replaced with a generic one; the errors, commands, and sequence of findings are otherwise as recorded.

How the job used SSH

An ETL service (etl.service) on the database server pulled data files from the application server over SSH, every cycle. It ran under systemd as an unprivileged service account, appsvc, and it didn't use the system's ssh command. It used Net::SSH, an SSH implementation written in Ruby and bundled with the vendor's tooling.

The servers had been moved to RHEL 8.8, whose OpenSSH server follows RHEL 8's system-wide crypto policy. That policy disables old algorithms, including the key exchange diffie-hellman-group1-sha1.

"Manual SSH works"

ETL alarms fired and no new records arrived. The obvious checks all passed: from a terminal on the database server, SSH to the application server worked, and restarting the ETL service changed nothing.

The team's notes capture why that first check was misleading:

Testing SSH from a shell proves nothing about an application that uses its own SSH library.

Four layers, one at a time

Each fix let the job get one step further, which revealed the next fault. The commands below are the ones recorded in the notes, with names anonymized.

systemctl status etl -l
journalctl -u etl -n 80 --no-pager
tail -f /opt/app/log/etl.log

Layer 1 · A compatibility patch that crashed the job. An earlier attempt to change the library's algorithms had been loaded into the service. After a restart, Ruby stopped before any ETL code ran:

alias_method': undefined method `kex_algorithms'

The patch tried to override an internal method that doesn't exist in Net::SSH 2.9.4. Overriding a library's internals ties you to one version of it. The team replaced it with an override at the library's public entry point, Net::SSH.start, which accepts a :kex option in every version:

# Force Net::SSH to avoid diffie-hellman-group1-sha1
require 'net/ssh'
module Net
  module SSH
    class << self
      alias __orig_start start
      def start(host, user, options = {}, &block)
        options ||= {}
        options[:kex] ||= [
          "curve25519-sha256",
          "curve25519-sha256@libssh.org",
          "ecdh-sha2-nistp256",
          "ecdh-sha2-nistp384",
          "ecdh-sha2-nistp521",
          "diffie-hellman-group-exchange-sha256",
          "diffie-hellman-group14-sha256"
        ]
        __orig_start(host, user, options, &block)
      end
    end
  end
end

The patch was loaded through a systemd drop-in rather than by editing the vendor's unit file, so a package update can't silently remove it:

sudo systemctl edit etl
#   [Service]
#   Environment="RUBYOPT=-r/opt/app/etc/net_ssh_kex_fix.rb"
sudo systemctl daemon-reload
sudo systemctl restart etl

Layer 2 · A key exchange the server refuses. Earlier failures had referenced the library's diffie_hellman_group1_sha1.rb. Left to its defaults, Net::SSH 2.9.4 could still try diffie-hellman-group1-sha1, which RHEL 8.8's sshd rejects during the handshake, before any user or key is checked. The :kex list above removed it. The SSH server's configuration was not changed: re-enabling a SHA-1 key exchange on the server would have weakened it for every client to fix one.

Layer 3 · A private key the library couldn't read. With the handshake working, the job failed inside Net::SSH::KeyFactory.read. The service account's key was in the newer OpenSSH private key format (it starts -----BEGIN OPENSSH PRIVATE KEY-----), which ssh reads but this version of Net::SSH doesn't parse reliably. The key was backed up and converted to PEM format in place:

cp -a /opt/app/.ssh/etl_key /opt/app/.ssh/etl_key.bak_$(date +%F_%H%M%S)
ssh-keygen -p -m PEM -f /opt/app/.ssh/etl_key -P "" -N ""

ssh-keygen -p rewrites the key file with the format given by -m, keeping the same key, so nothing needed changing on the server.

Layer 4 · The wrong remote user. The job still couldn't sign in. Adding debug logging of the connection parameters to the patch exposed the last fault in one line:

host=<application server> user=appsvc keys=["/opt/app/.ssh/etl_key"]

The job's configuration named a different remote account, the one that key was authorized for. But Net::SSH, like ssh, defaults the remote user name to the local user running the process, and this code path didn't pass the configured user through. The patch was extended to set the configured remote user explicitly for that host.

What confirmed the fix

• An SSH test through Ruby, run as the service account, succeeded.
• etl.service restarted cleanly.
• The ETL log showed a successful connection to the application server, with no further SSH, key, or algorithm errors.

Actions from the review

These come from the team's recorded recommendations:

1. Don't patch a library's internals. Override only its public entry points, such as Net::SSH.start.
2. Test SSH the way the application uses it, as the service account and through the application's own client, not from an interactive shell.
3. Standardize the format of automation keys to one the oldest client in use can read.
4. Pass the remote user explicitly in automation, never rely on the local user name.
5. Log the SSH parameters (host, user, key, algorithms) when a connection fails, so the next investigation takes minutes.

One wider lesson: the move to RHEL 8 didn't create these faults. It removed old defaults that had been hiding them.

Where this track teaches the skills

• Users, sudo & SSH: keys, authorized_keys, and why the server's log, not the client's, says why sign-in failed.
• systemd: Services, Timers & Logs: drop-in overrides instead of editing vendor unit files.
• Seeing Inside a Process: strace -f -e trace=network shows what an application's own SSH client is really doing.
• Security Hardening, AppArmor & TLS: fixing clients rather than weakening servers.

Pop quiz

Five questions: three on the ideas in this review, and two scenarios where you apply them. The order changes every time you take it, and 4 out of 5 passes.