Twenty-one active nodes, no shared SAN, zero downtime at peak load

Active-active clustering on storage that wasn't built to share

Oracle RAC gets its high availability from every node in the cluster reading and writing the same underlying storage at once: if one node fails, the others already have live access to its data and simply absorb its workload. That model assumes a shared SAN. AWS EBS volumes don't work that way: each volume attaches to exactly one EC2 instance at a time, which is the standard blocker that keeps most RAC deployments off public cloud entirely, or forces them onto more expensive dedicated-hardware options.

The platform this ran on also had an unusually spiky load profile: routine day-to-day traffic, punctuated by short, extreme surges (an examination-results release, an admissions deadline) where concurrent connections and transaction volume could jump by an order of magnitude with no warning window to scale into ahead of time. A cluster that could only fail over gracefully, but not sustain that kind of peak, wouldn't have solved the actual problem.

A software-defined storage fabric standing in for a SAN

                   [ Application tier / ERP / web services ]
                                     │
                     ┌───────────────┴───────────────┐
                     │                               │
              [ RAC node 01 ]     ...           [ RAC node 21+ ]
               (AWS EC2, r5)                      (AWS EC2, r5)
                     │                               │
                     └───────────────┬───────────────┘
                                     │
                     [ FlashGrid cluster fabric ]
                    (virtual shared-SAN over EBS)
                                     │
                     ┌───────────────┴───────────────┐
                     │                               │
           [ ASM diskgroup +DATA ]           [ ASM diskgroup +RECO ]
          (tablespaces, indexes)          (redo logs, archive, FRA)

Each node runs on memory-optimized EC2 instances with dedicated network interfaces separating client traffic, cluster administration, and the private cache-fusion interconnect RAC nodes use to coordinate with each other: that separation matters because cache-fusion traffic is latency-sensitive and shouldn't compete with application or management traffic on the same interface.

Verifying cluster health day to day comes down to a small set of commands run against Clusterware, the storage fabric, and ASM directly:

# Grid Infrastructure / Clusterware status
crsctl check cluster -all
crsctl status resource -t

# Storage fabric quorum and disk access
flashgrid-cluster status
flashgrid-node status

# ASM diskgroup throughput
su - grid -c "asmcmd lsdg"

Trade-offs

Running RAC yourself on EC2 with a third-party clustered-storage layer is a heavier operational commitment than a managed database service (every node, the storage fabric, and the patching cadence are yours to own, and that's a real ongoing cost against something like Aurora's operational simplicity. It was the right trade here because the workload's failure mode wasn't "an outage is inconvenient," it was "an outage during a results release affects a very large population of students and families with no way to reschedule the event") active-active RAC with zero-downtime patching bought headroom a simpler managed setup at the same scale wouldn't have.