Automating telecom failover, preserving licensed identity

A replacement host was not enough to restore service

Legacy recovery depended on hardware-oriented fencing and operator intervention. The telecom software also used a node-locked license tied to a MAC address and private IP. A newly provisioned host could therefore be healthy while its call-processing software remained unable to start with the expected licensed identity.

Treat recovery and licensing as the same workflow

A serverless Lambda recovery workflow was built to trigger from CloudWatch alarms. DynamoDB conditional writes coordinated orchestration attempts, while Systems Manager handled host-side network and service operations. A persistent secondary Elastic Network Interface (ENI) carried the identity registered with the software vendor.

The ENI approach preserved identity when that interface moved to a replacement host; it did not require assigning a custom MAC address or imply that ordinary instance reboots change interface identity.

Separate orchestration from the hosts being recovered

The orchestrator ran outside the two service nodes. Its responsibilities included acquiring a lease, moving the designated interface, requesting service changes, checking readiness and ownership, and handling the degraded host. These are distinct controls: a coordination lock does not itself fence a node or prove that only one node can accept writes.

The walkthrough illustrates the documented components. A safe recovery procedure must establish what prevents the previous owner from serving traffic or writing during promotion, including when an API call fails or a lease expires. The diagram is not proof of those guarantees.

Constrain the recovery mechanism

Recovery permissions should be scoped to the managed instances, interface, command documents, and lock resources where the AWS actions support resource-level restrictions. Command execution and orchestration logs provide an audit trail. The licensed interface still needs the appropriate service security groups and host permissions.

Exercise more than a clean instance stop

The source implementation describes staging exercises covering service failures, interface disruption, resource pressure, and stopped instances. Those exercises tested interface transfer and service recovery. Recorded timing should identify the failure mode, detection delay, start/end events, and retry behavior before being used as an availability commitment.

Same-AZ recovery is not regional disaster recovery

An ENI can attach only to an instance in the same Availability Zone. This workflow therefore cannot carry the same interface across an AZ or regional failure. Recovery elsewhere requires a separate network identity and a vendor-approved licensing approach. See the AWS interface attachment constraints.

Existing connections may drop during recovery and need client retry behavior. This design coordinates host-level recovery; it does not replace Aurora recovery or a complete regional failover plan.

Recovery included the application's licensed identity

The work combined host recovery with interface movement and application checks, addressing a dependency that a generic instance replacement would miss. The portfolio does not claim zero split-brain risk or a universal sub-minute recovery time; those depend on failure conditions and retained test evidence.

Document failure boundaries and recovery evidence

Capture lease-expiry tests, partially completed interface moves, unavailable Systems Manager agents, and failed fencing calls. Verify the promotion/fencing sequence and record the recovery timeline for each scenario, including the point at which clients can successfully use the service.