Tips For Local Beiwe Development

Before anything else...

You need to create a virtual environment for running the Beiwe codebase. This is a detail requirement of Python software development that you have to get comfortable with them. Virtual environments allow you to have separate collections of Python libraries installed for different projects.

Beiwe is a living project, consequently it periodically requires updates to the underlying version of Python. To that end I personally prefer pyenv and pyenv-virtualenv for managing my various versions of Python and their virtual environments. There is a convenient install that command that you can find here, you can find the project's repo here, you can find the list of system dependencies required to compile Python here, and the common build issue help page here.

Running Locally

You can create a file that defines the Beiwe environment variables, and then source it before running the local server or ipython shell (accessible via python manage.py shell_plus). The repo is pre-configured to ignore everything in a folder named private in the root of the repo, so go ahead and create a file named something like env.sh. Files in this folder will be ignored by git.

(You can also mimic the data processing server's mechanism and populate a config/remote_db_env.py file.)

For variable names follow the example config/remote_db_env.py.dummy file for your

import os
os.environ['SENTRY_ANDROID_DSN'] = 'https://foo:[email protected]/foo'
os.environ['SENTRY_ELASTIC_BEANSTALK_DSN'] = 'https://foo:[email protected]/foo'
os.environ['SENTRY_DATA_PROCESSING_DSN'] = 'https://foo:[email protected]/foo'
os.environ['FLASK_SECRET_KEY'] = 'replace_with_random_string'
os.environ['SENTRY_JAVASCRIPT_DSN'] = 'https://[email protected]/foo'
os.environ['SYSADMIN_EMAILS'] = 'webmaster@localhost'
os.environ['DOMAIN_NAME'] = 'beiwe.org'
os.environ['RDS_PASSWORD'] = 'password'
os.environ['RDS_USERNAME'] = 'beiweuser'
os.environ['RDS_DB_NAME'] = 'beiweproject'
os.environ['RDS_HOSTNAME'] = 'localhost'
os.environ['S3_BUCKET'] = 's3_bucket_name'
os.environ['BEIWE_SERVER_AWS_ACCESS_KEY_ID'] = 'somestring'
os.environ['BEIWE_SERVER_AWS_SECRET_ACCESS_KEY'] = 'somestring'

(RDS is AWS' database service, that's where the database connection credentials go.)

The Database

You have to run a local instance of PostgreSQL, the exact version is flexible.

• If you are on a Mac check out https://postgresapp.com/ (Running Postgres via Homebrew successfully and reliably can be a real pain.)
• If you are on Linux distribution or WSL, follow a normal guide online.
• Set up your database's name, password etc. and add them to those local environment variables.

Django's test runner occasionally barfs and corrupts itself under certain (avoidable with experience) circumstances. You may need to run this Django management command to clear out the database:

python manage.py drop_test_database

Database is up, what next?

• You always start with a source private/env.sh, but you can skip that if you went with the remote_db_env.py file.
• To run the local server run python manage.py runserver from the project root.
• If critical variables are not set you should see an explicit message stating what variable is missing or invalid.
• If you immediately hit a Python stack trace instead of a helpful message .... well that would be bad. If you encounter this and it was the result of bad or out-of-date documentation please submit a bug report.

If this is your first-ever run (or if you pulled down a new version with database schema changes) you will see a message about unapplied migrations.

• migrations are code you must execute once per database schema change, they.... update the database schema.
• migrations are stored in database/migrations
• run the command python manage.py showmigrations to view your current state migrations
• run the command python manage.py migrate to run all migrations up to the most recent one
• run the command python manage.py migrate some_migration_name to roll back the database schema to a specific target schema

Migrations are a critical concept inside of Django, which is includes this ORM.

One of the migrations sets up a default admin named "admin" with a password of abcABC123!@#.

(submit a bug report if this is incorrect, documentation matters.)

The Python and Django shell with database access

This project has a well configured shell environment to assist everyone in development and debugging. This shell is present on any server you may ssh onto via the same command explained below. This is the way you should be interacting with live data, if you do that at all.

To access the shell run python manage.py shell_plus from the project root (this may prompt you to provide variables).

• This command will drop you into a special ipython shell environment with a number of Extremely Useful details already important.
• This shell uses iPython and has tab autocompletion.
• All database models (tables) are automatically imported
• We have added quite a few useful extra imports over in config/django_settings.py
• The libs.utils.shell_utils file is imported. Among other tools, it includes shorthand mechanisms to get participant and researcher (website users) objects.

Running Data Processing Manually

You may need to run the data processing code manually, this can occur if you have a specific participant uploading so much data it crashes your server, forcing you to disable celery while you debug. (Note: data processing code has seen a lot of memory optimization effort, this scenario is unusual and implies a bugged participant that is running non-stop recording sessions. Such a participant should be disabled via the Permenantly Retire button on their Participant Page. Under normal circumstances 4GB of ram should be more than sufficient for even the most aggressive study settings.)

Load the shell

python manage.py shell_plus
In [1]: p = P(participant_id)
In [1]: from libs.file_processing_core import easy_run
In [2]: easy_run(p)