MECO API

JSON API for account, billing & key management services. Aimed to moderate explicit content online 🔞.

Getting Started

These instructions will get you a copy of the project up and running on your local machine for development and testing purposes. See deployment for notes on how to deploy the project on a live system.

Project structure

The project folder structure is shown below.

• dev-scripts: useful exported PostgreSQL scripts for development.
• postman: exported v2.1 Postman JSON collection file for testing.
• src: contains project source code.

Prerequisites

Make sure you have installed all of the following prerequisites on your development machine:

• Download & install Git. OSX and Linux machines typically have this already installed.
• Install the latest version of Java.
• Rather than installing Maven manually you may use the Maven Wrapper mvnw or mvnw.cmd.
• An IDE plugin for Project Lombok such as this.
• You may need to set your JAVA_HOME environment variable.
• A PostgreSQL database for storing account, billing and key data.
• A Stripe account with access to your secret key for processing payments.
• A Mailgun account with access to your secret key for sending emails.
• Optional: An API testing tool such as Postman for exploratory API testing.

The recommended way to get the MECO API is to use Git to directly clone the repository:

# clone repository
git clone https://github.com/the-pragmatic-dev/meco-api.git
cd meco-api/

Default properties found in src/main/resources/application.yml will need to be updated. You must ignore local updates to application.yml to protect your secret keys. Mark the file as so: git update-index --skip-worktree src/main/resources/application.yml. Either set environment variables using the export command or pass them to the application at runtime:

# set environment properties and run service
export DATASOURCE_URL=jdbc:postgresql://{host}:{port}/{database}
export DATASOURCE_USERNAME=username
export DATASOURCE_PASSWORD=password
...
./mvnw spring-boot:run

# or set properties at runtime
./mvnw spring-boot:run -Dspring-boot.run.arguments= \
  --logging.file.path={path}, \
  --server.port={port}, \
  --spring.datasource.url={url}, \
  --spring.datasource.username={username}, \
  --spring.datasource.password={password}, \
  --spring.jpa.show-sql=false, \
  --spring.flyway.clean-on-validation-error=false, \
  --spring.flyway.locations=classpath:/db/migration,classpath:/db/data/prod, \
  --security.jwt.token.secret-key={key}, \
  --security.jwt.token.expire-length=300000, \
  --stripe.secret-key={key}, \
  --mailgun.secret-key={key}

Spring JPA is set to only validate the DDL - database configuration management is done through Flyway. Flyway will configure the database schema on startup by running all migration files under src/main/resources/db/migration. By default, a dummy admin account is created with the following credentials: username: [email protected], password: password.

GeoLite2 database

The application reads from a GeoLite2 city database in order to verify if our users are logging in from a new device/location. To retrive the database, you will need to:

1. Create a MaxMind account.
2. Generate a new licence key.
3. Update the geolite2.permalink application property with your licence key.

On startup the application will fetch the database and cache it in the geolite2.directory. Ensure you have ran the following script prior to application startup which creates the database directory and sets the owner / group:

./scripts/init.sh /var/lib/meco/ johnsnow

To test the device / geolocation functionality, ensure requests contain the following headers:

Key	Value
X-FORWARDED-FOR	196.245.163.202
User-Agent	User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36

Installing

From the cloned workspace compile the project which will also run all unit tests:

# compile and run
./mvnw install

the output from a successful build is below.

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  2.001 s
[INFO] Finished at: 2020-04-01T20:50:20+01:00
[INFO] ------------------------------------------------------------------------

Running the tests

Maven profiles exist for running either the unit tests or integration tests, unit and integration respectively. The unit profile is active by default. To run both the unit and integration tests use the all profile.

Unit tests

You can run unit tests by running the following command at the command prompt:

./mvnw clean test

Integration tests

You can run integration tests by running the following command at the command prompt:

./mvnw clean verify -P integration

To run an individual integration test:

./mvnw -Dit.test=AccountEndpointIT#shouldReturnLatestSecurityLogs clean verify -Pintegration

Sonar report

Sonar properties are defined in pom.xml. Test reports are pushed to SonarCloud using the following command on our Travis CI build server:

./mvnw sonar:sonar

JaCoCo generates individual coverage reports for unit and integration tests. It also generates an aggregate report called aggregate.exec, this is pushed to SonarCloud.

Coding style guide

The Maven Checkstyle plugin will check for violations before compiling. Code that does not adhere to the Google Java Style Guide as defined in checkstyle.google.xml will cause the build to fail.

Deployment

Terraform is used for provisioning infrastructure through code. We use it to create a our DigitalOcean environment.

Prerequisites

1. Install Terraform CLI (version >= 0.12) and add it to your PATH.
2. Generate SSH keys and add to Digital Ocean. On the menu Account > API > Add SSH key . After you’ve added, it will show you a fingerprint that you will need to run the terraform command.
3. Generate a DigitalOcean API token. Menu Manage > API > Generate New Token.

Terraform

Terraform files are located under the terraform directory. In order to run them, you need to pass the variables as params (or Terraform will ask one by one). You can configure env vars as follows:

export DO_TOKEN=xxx
export SSH_FINGERPRINT=xxx

Also we need to call init to download the provider plugins:

terraform init

And then finally apply it:

terraform apply \
  -var "do_token=$DO_TOKEN" \
  -var "pub_key=$HOME/.ssh/id_rsa.pub" \
  -var "pvt_key=$HOME/.ssh/id_rsa" \
  -var "ssh_fingerprint=$SSH_FINGERPRINT"

It first shows the plan that is about to be executed. If the plan looks ok, type yes (or use --auto-aprove). The environment will then be built.

For development purposes you can destroy the newly created environment by running destroy:

terraform destroy \
  -var "do_token=$DO_TOKEN" \
  -var "pub_key=$HOME/.ssh/id_rsa.pub" \
  -var "pvt_key=$HOME/.ssh/id_rsa" \
  -var "ssh_fingerprint=$SSH_FINGERPRINT"

API documentation

TODO api documentation to be generated by Slate.

Key generation

Sensitive fields in domain models are set to be ignored on JSON serialisation by default, such as account password hashes and the actual API key. To allow the key to be viewed only once by the user on creation, Programmatic JSON Views are used to include selected fields on POST requests.

Authorization

JWT is used for authorization. JWTs are signed using a secret key. Once a user is logged in, each subsequent request will require the JWT, allowing the user to access routes, services, and resources that are permitted with that token. Below is an example of authorizing with the API and using the JWT in a subsequent request:

# signin to service
curl -X POST "http://localhost:8080/accounts/signin" -H "accept: */*" -H "Content-Type: application/json" -d "{\"username\":\"[email protected]\",\"password\":\"password\"}"

# send JWT (token) with new request which lists all API keys
curl -X GET "http://localhost:8080/api-keys" -H "accept: application/json" -H "Authorization: Bearer {token}"

JWT claims contain both the account username and given roles.

Travis CI

The following environment variables need to be set within Travis CI:

Name	Value
GEOLITE_URL	https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-City&license_key={YOUR_LICENSE_KEY}&suffix=tar.gz
PERSPECTIVE_SECRET_KEY	{YOUR_PERSPECTIVE_SECRET_KEY}
SONAR_TOKEN	{YOUR_SONAR_TOKEN}
STRIPE_SECRET_KEY	{YOUR_STRIPE_SECRET_TEST_KEY}

Monitoring

Actuator endpoints are enabled by default to monitor and interact with the running service.

• endpoints: http://{host}:{port}/actuator/.
• health: http://{host}:{port}/actuator/health.
• info: http://{host}:{port}/actuator/info.
• flyway: http://{host}:{port}/actuator/flyway.
• metrics: http://{host}:{port}/actuator/metrics.
• metrics-name: http://{host}:{port}/actuator/metrics/{requiredMetricName}.

Built With

• Spring Boot - Java-based framework.
• Project Lombok - Java library to minimize boilerplate.
• Stripe Java - Java library for the Stripe API.
• Mailgun Java - Java library for the Mailgun API.
• GeoLite2 - Geolocation database for verifying user locations.
• Programmatic JSON Views - Java library to include selected JSON fields.
• Checkstyle - Java tool for checking source code adheres to Google code standard.
• JWT - JSON-based access tokens that assert a number of claims.
• Maven - Java dependency management.
• Slate - Provides API documentation.
• PostgreSQL - Relational database.
• Flyway - Database configuration management.

Documentation

• Account service - Management of a users account.
• Auth service - Management of access and refresh tokens.
• Billing service - Management of account subscriptions.
• Email service - Management of account email triggers.
• Key management service - Management of creating, updating and deleting API keys.

Contributing

Please read CONTRIBUTING.md for details on our code of conduct, and the process for submitting pull requests to us.

Versioning

We use SemVer for versioning. For the versions available, see the tags on this repository.

Authors

• Stephen Cathcart - Initial work - The Pragmatic Dev Ltd

See also the list of contributors who participated in this project.

License

MECO API is licensed under the Apache License, Version 2.0. See LICENSE for the full license text.

Acknowledgments

• Hugo Firth - Tech consultant - GitHub