A four-to-eight-hour manual recovery, reduced to one script
A single EC2 instance hosting enterprise mail was a real single point of failure, and recovering it by hand) reassigning the IP, finding and reattaching the right volume, remounting, restarting: routinely took the better part of a working day. This case study covers automating that entire sequence into one script, plus the backup and DNS-authentication hardening that went with it.
At a glance
- Problem
- Enterprise email ran as a single EC2-hosted instance handling a high volume of daily communications across administrative offices, campuses, and external partners (a real single point of failure. Recovering from an instance impairment meant a manual sequence of reassigning the elastic IP, locating and reattaching the data volume, remounting it, and restarting services by hand, which took four to eight hours end to end. On top of that, missing email-authentication records meant a meaningful share of outbound mail was being flagged as spam before it ever reached recipients.
- Approach
- A single Bash script that automates the entire failover sequence end to end) instance metadata lookup, elastic IP reassociation, EBS volume attachment with a polling wait loop, filesystem mount (including NVMe device-mapping for newer instance types), and a service restart (plus a tiered S3 backup pipeline and full SPF/DKIM/DMARC DNS hardening.
- My contribution
- Lead infrastructure & disaster recovery architect
Key results
A single instance, a manual runbook, and hours of downtime
The mail platform ran on one EC2 instance. When that instance became impaired (a host-level hardware issue, a failed health check) recovery meant a human working through a checklist: figure out which elastic IP needed reassigning, find the correct data volume among everything else in the account, attach it to a new instance, wait for the attachment to actually complete, work out which block device it landed on (which can vary depending on instance type), mount it, and finally restart mail services and confirm every daemon came back healthy. Done carefully, that took four to eight hours: and every one of those hours was mail fully down for the organization.
Separately, the mail platform had a deliverability problem: without properly configured SPF, DKIM, and DMARC records, a meaningful share of legitimate outbound email was landing in spam folders or getting flagged by recipient mail servers.
One script replaces the entire manual sequence
[ External mail gateways / Internet ]
│
[ Elastic IP — reassociated ]
│
┌─────────────────┴─────────────────┐
│ │
[ Primary EC2 host ] [ Standby EC2 host ]
(active instance) (failover target)
│ │
└──────► [ Data EBS volume ] ◄───────┘
(reattached on failover)
│
[ Nightly / weekly sync ]
▼
[ AWS S3 backup target ]Implementation notes
- The script does exactly what the manual runbook did, in order, without the waiting-and-guessing. It queries EC2 instance metadata for the new host's instance ID, uses the AWS CLI to atomically reassociate the elastic IP, checks whether the data volume is already attached and attaches it if not, then polls the AWS API in a loop until the volume actually reports an
attachedstate rather than assuming a fixed delay is long enough. - It handles the one thing that trips up manual recovery under pressure: device naming. Newer, Nitro-based EC2 instance types expose attached EBS volumes as NVMe devices rather than the traditional
/dev/xvdf-style path, so the script checks for the expected device and falls back to the NVMe mapping automatically instead of leaving that judgment call to whoever's running the recovery at 2am. - Recovery ends in a verified state, not just a restart command. After mounting the filesystem, the script restarts mail services and then explicitly checks that every core daemon (directory, mail transport, mailbox store) reports healthy, rather than treating "the restart command didn't error" as success.
- Backups are tiered, not a single nightly dump. A full backup runs weekly and incremental backups run on the other days, each compressed and uploaded to a dedicated S3 backup bucket, with a rolling 7-day local retention window so recovery doesn't depend on how far back the most recent full backup happens to be.
- Deliverability was a DNS problem as much as an infrastructure one. Publishing 2048-bit DKIM keys, a strict SPF record, and a DMARC policy requiring authentication on all outbound mail took the deliverability rate from 82% to 99.8%: the infrastructure automation solved availability, this solved whether the mail actually arrived once it was sent.
# Automated failover — run on the replacement instance
./ec2_zimbra_failover.sh
# What it does, in sequence: detects the private IP, reassociates the
# elastic IP, attaches (or confirms) the data volume, mounts it — with
# NVMe fallback — and restarts mail services with a health check.Trade-offs
This is a single-instance failover pattern automated end to end, not a true multi-AZ active-active mail deployment: the mail platform is still down for the seconds-to-minutes the script takes to run, just not for hours. Building genuine multi-AZ high availability into an already-established mail deployment would have meant a much larger migration project than was in scope here; automating the existing recovery path to the point where it reliably runs in under three minutes was the trade that fit the actual constraint, and the same "detect, reassociate, remount, verify" pattern generalizes cleanly to any other stateful EC2 service that doesn't have built-in HA.
Questions about this case study?
Happy to go deeper on the failover script's polling logic, the tiered backup retention design, or the DNS-authentication setup.
anupojuprudhvi@gmail.com