Upgrading from Python 2 to 3

Instructions for upgrading a scalable Beiwe cluster deployment from Python 2.7 to 3.6

This document is very old

Python 2.7 was end-of-life'd on January 1st, 2020, and Python 3.6 on 2021-12-23. This page is around for reference only.

If you need to update a very old version of the beiwe-backend you should probably run a manual database dump and restore onto a newly deployed Elastic Beanstalk Beiwe cluster.

Setup

Before anything else you will need to set up Python 3. 3.6 is the target version, although newer versions should work just fine.

Essentially all OSes will have a Python 3 runtime available through the usual channels (or already present). The executable may be named python3 in which case you will have to substitute that command for uses of python on your local machine. You may also have to install pip3. (This is the case on Ubuntu, which the manager/worker servers run.)

You should already have the Elastic Beanstalk Command Line Interface (awsebcli) configured; you can find documentation here: https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/eb-cli3.html

Upgrading the deployment to Python 3 will require the installation of AWS Command Line Tools (awscli). You can find documentation here: https://aws.amazon.com/cli/

OBNOXIOUS NOTE: there is (or was) a requirements incompatibility between awscli and awsebcli. The easiest fix is to just install each one via the pip command as needed. You will only need awscli briefly. (There are more extended solutions to the problem not described here.)

Steps:

1. Go into the AWS online console and terminate any manager and worker servers you have running.

2. Check out onto the downtime branch of the Beiwe Backend repo. This is a branch that is safe to use on Python 2 and 3 runtimes, and can generally be used whenever you have downtime needs. (This branch may be updated periodically, so you should pull whenever you deploy it. This is just basic git stuff.)

   git pull or git fetch

   git checkout downtime

   Using awsebcli deploy this branch to your target cluster.

   eb deploy [appropriate-cluster-name-here]

   You will know that downtime mode is active when the website states it is in downtime mode (load any page). Note that you may get a Severe status warning due to a high proportion of 400 or 500 errors. This is unavoidable, it is due to a design choice made back in 2014; we need to send the mobile app 400 or 500 errors or else it will think it's upload attempts completed successfully, otherwise it will delete the "uploaded data". We will try and update the

3. Install awscli (the AWS Command Line Interface).

   The install command is pip install awscli or pip3 install awscli. After a successful install you should have access to a new command line tool which is simply aws.

4. Configure awscli Run aws configure and enter the aws credentials and the relevant region. (I also recommend entering "json" in the response type preference, this is just for consistency in the future.)

   At this point you will have a command line interface to manipulate your AWS account with all the permissions of the credentials that you just gave it. A credential configuration file will be deposited in ~/.aws.

5. Use awscli Run this command to get the exact name of the current Python 3 platform "solution stack":

   aws elasticbeanstalk list-available-solution-stacks --output=json --query 'SolutionStacks' | grep "running Python 3.6"

   You may get several lines of output. Copy the newest version of the Solution Stack, include the quotation marks, but clear out any trailing and preceding whitespace characters and commas. It will look something like this, but could be different depending on your region and the current date: "64bit Amazon Linux 2018.03 v2.9.2 running Python 3.6"

6. The conversion.

   This command will take your Python 2 platform and replace it with the Python 3 platform. You will see old servers get killed and new ones spun up.

   Be aware that the operation will provide a formatted response but does not block until the process is finished. It will appear to take about 3 seconds, but you should check the status of the cluster and wait for the downtime message of the website to appear again.

   aws elasticbeanstalk update-environment --environment-name "YOUR_CLUSTER_NAME" --region APPROPRIATE_REGION_SHORTNAME --solution-stack-name "THE_SOLUTION_STACK_NAME_IN_QUOTES"

7. Bring the site back up. Use git to check out onto the production branch and then use awsebcli (remember, it may require a reinstall) to deploy it. Make sure you are on the most recent version of the production branch.

   eb deploy [appropriate-cluster-name-here]

8. Create new data processing workers.

   Run launch_script.py to create a new Data Processing Manager server, note that it now also uses Python 3.

Note: For the Python 3 version of Beiwe the worker servers are optional. The manager server will now both manage any workers and act as a worker server itself. Create a worker server(s) if the manager server on its own cannot keep up with your uploaded data, or you can opt to terminate and then deploy a bigger manager server by changing MANAGER_SERVER_INSTANCE_TYPE in your environment server settings.

You should now be up and running beiwe on Python 3!