Scheduler Migration Guide

Migrating an app to the CloudFormation backend

This document will guide you in the steps necessary to safely migrate applications to the new CloudFormation backend, with minimal amount of downtime.

Step 1 - Create the CloudFormation stack

The first step will create the new CloudFormation stack with the ECS/ELB resources created, but without altering the <app>.empire CNAME record.

To perform the first step, run:

$ emp set EMPIRE_SCHEDULER_MIGRATION=step1

Then wait for the new CloudFormation stack to finish creating: https://console.aws.amazon.com/cloudformation/home

If the stack creation is rolled back, take note of the error in the stack events, fix the issue (usually Empire is misconfigured, or you haven't given Empire the necessary permissions), then destroy the stack, and run step 1 again:

$ emp set EMPIRE_SCHEDULER_MIGRATION=step1

Step 2 - Swap DNS

You have two options:

1. Manually remove the <app>.empire CNAME and then proceed to step 3.
2. Manually remove the <app>.empire CNAME, then set the DNS parameter to true in the newly created CloudFormation stack. Use this if you want to test the new stack before completely removing the old ECS/ELB resources.

If you're running a high load service, you may need to contact AWS support to pre-warm the new ELB.

Step 3 - Remove the old ECS/ELB resources

This will remove the old ECS services, and ELB's.

$ emp set EMPIRE_SCHEDULER_MIGRATION=step2

Step 4 - Finalize the migration

$ emp unset EMPIRE_SCHEDULER_MIGRATION

Downtime

During the migration, the only downtime that should occur is within step 2, when DNS is swapped from the old load balancer to the new load balancer. This downtime should be minimal (if any) since both the old load balancer and the new load balancer will be accepting traffic. If absolute zero downtime is needed, you can point applications directly at the ELB hostname temporarily until the new CNAME is resolvable.

Rollback

If for whatever reason, you need to roll back to the old scheduler, that can be done easily by simply removing the CloudFormation stack, then restarting the application.