Skip to content
John Wieczorek edited this page Feb 21, 2019 · 6 revisions

Developing

Python (2.7.5) App Engine Python (1.9.10) nodejs (0.10.31) Git

On the server side, VertNet runs on the Google App Engine Python 2.7 runtime. To develop, you will need to download and install the latest Python SDK and add it to your PATH. Development has been tested with Python 2.7.5 and App Engine 1.9.10.

On the client side the web application uses nodejs. Development has been tested with nodejs 0.10.31.

To manage updates to the Github repository, use Git.

Getting started

Make sure you have Git installed, and then from the command line:

$ git clone [email protected]:VertNet/webapp.git

That will download the full code repository into a directory named webapp.

Since the webapp runs on Google App Engine, we can use the local development server for testing changes before deploying them to production on App Engine. First we'll need some test data. Go into thewebapp directory, download this file, and unzip it. Then at the command line:

$ dev_appserver.py --storage_path=data app.yaml

The webapp should now be running locally at http://localhost:8080 and you get an admin console at http://localhost:8000/.

Although the webapp will run locally, it still requires the JSON authentication file named 'auth.txt' to access it. A sample of this file is included in the code base (https://github.com/VertNet/webapp/blob/master/auth.txt.sample) and will needed to be customized with your information and google key.

Deploying

Application deployment begins with having new web site content or capability that needs to be viewed or tested. Usually this is done by developing a well-scoped feature in a branch of the code, and deploying that feature to a test version of the application on AppEngine. Once a feature is tested, it can be merged with the main branch and deployed to the default production web application (prod) at vertnet-portal.appspot.com, where it is publicly accessible via portal.vertnet.org. A list of the versions for the vertnet-portal application can be accessed via the Versions list on the Google App Engine dashboard at:

https://appengine.google.com/dashboard?&app_id=s~vertnet-portal

To develop a new feature called 'newfeature' off of the master branch:

git checkout master
git pull
git checkout -b feature/newfeature

Make changes to the code and commit these to the branch in the repository.

git commit -a -m "Comment on my changes."
git push origin feature/newfeature

To test the changes in a particular version of the application (myversion in this example), compile the code by running nodejs in the tools directory, then deploy the application to App Engine by running appcfg.py in the webapp directory. Note: If appcfg.py is not found from the command line, create symbolic links using GoogleAppEngineLauncher. Note: If nodejs is not already installed, download and install it and make sure it is accessible from the system path.

cd tools
node r.js -o build.js
cd ..

NOTE: The appcfg.py method of deployment is out of date. Use gcloud instead. A file portal-web.yaml has been added to the repository for this purpose. Edit this file to set the service location (default or portal-web). For detailed deployment instructions using this new method, refer to https://github.com/VertNet/portal-web/wiki/Portal-development#deployment

appcfg.py update -V myversion . 

Deploy a non-production version:

gcloud app deploy portal-web.yaml --version pagodarose --no-promote

Deploy a production version:

gcloud app deploy portal-web.yaml --version prod --promote

You will be asked to enter a gmail address and password that has administrative rights to the application. To see if you have these rights, open the project dashboard at the following URL:

https://appengine.google.com/dashboard?&app_id=s~vertnet-portal

If the deployment goes well, there should be no reported errors, and the output from appcfg.py should be something like this:

02:13 PM Cloning 94 static files.
02:13 PM Cloning 377 application files.
02:13 PM Uploading 3 files and blobs.
02:13 PM Uploaded 3 files and blobs
02:13 PM Compilation starting.
02:13 PM Compilation completed.
02:13 PM Starting deployment.
02:13 PM Checking if deployment succeeded.
02:13 PM Deployment successful.
02:13 PM Checking if updated app version is serving.
02:13 PM Completed update of app: vertnet-portal, version: gittest
02:13 PM Uploading index definitions.
02:13 PM Uploading task queue entries.

After deployment, the version should be accessible at a URL based on the version name given:

http://[myversion].vertnet-portal.appspot.com

If the application is not running, look in the App Engine Console log for errors. If there is an error such as

Traceback (most recent call last):
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 239, in Handle
    handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/lib_config.py", line 353, in __getattr__
    self._update_configs()
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/lib_config.py", line 289, in _update_configs
    self._registry.initialize()
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/lib_config.py", line 164, in initialize
    import_func(self._modname)
  File "/base/data/home/apps/s~vertnet-portal/tuco.373402616596125535/appengine_config.py", line 53, in <module>
    auth = get_auth()
  File "/base/data/home/apps/s~vertnet-portal/tuco.373402616596125535/appengine_config.py", line 32, in get_auth
    auth = json.loads(open(path, "r").read())
IOError: [Errno 2] No such file or directory: '/base/data/home/apps/s~vertnet-portal/tuco.373402616596125535/auth.txt'

you are likely missing the auth.txt file in the webapp folder, which takes care of authentication. A sample of the content of this file is given at https://github.com/VertNet/webapp/blob/master/auth.txt.sample. There is also an authentication file for CartoDB that is needed to connect with a CartoDB instance for information about publisher statistics, writing query statistics, and rendering maps. A sample of the content of this file, cdbkey.txt is given at https://github.com/VertNet/webapp/blob/master/vertnet/service/cdbkey.txt.sample.

Once satisfied that a feature is ready for incorporation in the production application, make sure that you are in the branch, that it is up-to-date with any other changes that might have been committed:

git checkout feature/newfeature
git pull
git commit -a -m "Comment on my changes."
git push origin feature/newfeature

Next, create a pull request for the feature, which is a request to have it reviewed and merged into the master branch if it passes review. In GitHub, go to the application at

https://github.com/VertNet/webapp

There will be a list of recently pushed branches, each with a "Compare & pull request" button. Click on the button for the branch you want merged. The resulting page allows you to review the changes, enter comments and send the pull request by clicking on the "Send pull request" button. Someone else should always review the pull request in best practice, and merge if the request looks sound. Merging can be done by looking at the pull request page, where there is a button to "Merge pull request" if there are no outstanding conflicts. The reviewer can make comments on this page if something needs to be changed, or comment and merge the pull request, saying what issue(s) it resolves.

Once a branch is no longer needed, it can be removed using the following command:

git branch -D feature/newfeature
git push origin --delete feature/newfeature 

Alternatively, you can delete the remote origin branch on GitHub on the application's branches list page:

https://github.com/VertNet/webapp/branches

Start the next feature by checking out master (or another branch in progress and following the same development pattern.

Deploy the production version

Once the app is satisfactorily tested, deploy it to the production version:

$ gcloud app deploy --version prod portal-web.yaml --promote