This application is in charge of providing the core APIs for Grid'5000.
The project is hosted at https://gitlab.inria.fr/grid5000/g5kapi.
Please send an email to [email protected] if you cannot access the code, but if you read this, it's normally good…
The app is packaged for Debian jessie and Debian stretch. Therefore installation is as follows:
$ sudo apt-get update
$ sudo apt-get install g5k-api
Most configuration file for the application are those in /opt/g5k-api/config. They are linked from /etc/g5k-api, but should the link be broken, the application will use the version /opt/g5k-api/config.
If the database is not configured when the package is installed, you'll need to run migration after it has been setup
$ sudo g5k-api rake db:migrate
The service's execution environment is built by the /usr/bin/g5k-api script, that will source any .sh file in /etc/g5k-api/conf.d/ before calling /opt/ug5k-api/bin/g5k-api. This is how bundled gems are setup before execution, and SECRET_KEY_BASE is set for production environment.
You can check the installation by running a few commands to gather information about the application's execution environment
$ sudo g5k-api rake db:migrate:status
$ sudo g5k-api rake about
$ sudo g5k-api console #this will drop you in the rails console
$ sudo g5k-api rake dbconsole #this will drop you in the database's console
- The
masterbranch is the main development branch, from which stable release are generated and pushed to api-server-devel first, and then api-server-v3, on Grid'5000 - New features and fixes are expected to be developped in specific branches, and submitted for inclusion using Merge Requests.
-
This app comes with a Vagrant box used for development, testing and packaging.
By default, the vagrant box will provision a proxy, to get access to the live status of sites and to the home directory of users, except on one site where status will be served locally through a tunnel to that site's oardb.
For users with a working installation of vagrant and virtualbox, setting up a working environement starts with a simple
$ DEVELOPER=dmargery OAR_DB_SITE=rennes vagrant up --provision $ vagrant ssh vagrant> cd /vagrantThe vagrant provisionning script will attempt to configure the VM's root and vagrant accounts to be accessible by ssh. By default, it will copy your authorized_keys, but you can control the keypair used with SSH_KEY=filename_of_private_key
And as the application relies on external data sources, you'll need to connect it with a reference-repository, an OAR database, a kadeploy3 server, and a jabber server to exercice all its functionnality, in addition to its own backend services that are already packaged in the Vagrant box. As this is not trivial a requires some compromises, the development of this application relies strongly on unit tests.
A useful set of ssh tunnels is created with
vagrant> bundle exec rake tunnels:setup
-
Setup the database schema:
vagrant> bundle exec rake db:setup RAILS_ENV=development -
Give access to reference data to expose. The scripts expect you have a checkout version of the reference-repository in a sibling directory to this code that is mounted on /home/vagrant/reference-repository in the vagrant box.
(g5k-api) $ cd .. (..) $ git clone ssh://[email protected]/srv/git/repos/reference-repository.git reference-repositoryYou might not have admin access to Grid'5000's reference repository. in this case, you could duplicate the fake repository used for tests, in the spec/fixtures/reference-repository directory.
(g5k-api) $ cp -r spec/fixtures/reference-repository .. (g5k-api) $ mv ../reference-repository/git.rename ../reference-repository/.gitDo not attempt to use the directory directly, as unit test play with the git.rename dir.
-
Get access to a OAR database, using one of the two methods described hereafter:
-
Get your hands on a copy of an active database, and install it. Don't worry about the error messages when seeding the development database: most of them come from the fact that the database is empty and therefore the drop statements fail.
$ export OAR_DB_SITE=rennes $ ssh oardb.$OAR_DB_SITE.g5kadmin sudo cat /var/backups/dumps/postgresql/oar2.sql.gz > tmp/oar2_$OAR_DB_SITE.sql.gz $ gunzip tmp/oar2_$OAR_DB_SITE.sql.gz vagrant>SEED=tmp/oar2_rennes.sql RAILS_ENV=development bundle exec rake db:oar:seed -
Or tunnel your way to a live database (as g5k-api only requires read-only access) This is particularly usefull if you want to develop on the UI (but with bad site information). You should setup an SSH tunnel between your machine and one of the oardb servers of Grid'5000, so that you can access the current jobs:
$ # In an other shell, create a tunnel from the vagrant machine to Grid'5000 $ # (done as part of bundle exec rake tunnels:setup) vagrant> ssh -NL 15433:oardb.rennes.grid5000.fr:5432 access.grid5000.fr $ # edit the development section of config/defaults.yml to oar: <<: *oar host: 127.0.0.1 port: 15433 username: oarreader password: read database: oar2
-
-
As a reminder you will need to activate ssh tunnels for the proxy to work fully. This requires that you ssh keys were loaded into ssh agent befor you connected to vagrant.
vagrant> bundle exec rake tunnels:setup -
To run the server, just enter:
$ RAILS_ENV=development bundle exec ./bin/g5k-api rails server -
If you need to be authenticated for some development, use:
$ RAILS_ENV=development HTTP_X_API_USER_CN=dmargery WHOAMI=rennes sudo -E bundle exec ./bin/g5k-api rails server -e development
That's it. If you're not too familiar with rails, have a look at
http://guides.rubyonrails.org/.
You can also list the available rake tasks to see what's already automated for you:
$ bundle exec rake -T
Assuming you have your development environment setup, the only missing step is to create a test database, and a fake OAR database.
-
Create the test database as follows:
$ RAILS_ENV=test bundle exec rake db:setup -
Create the fake OAR database as follows:
$ RAILS_ENV=test bundle exec rake db:oar:create # provisioned by puppet $ RAILS_ENV=test bundle exec rake db:oar:seed -
Launch the tests:
$ RAILS_ENV=test bundle exec rspec spec/ # or `bundle exec autotest` if rake fails (issue with Rails<3.1 and em_mysql2 adapter with evented code). -
Update reference-repository used for tests
Data used for tests live in the spec/fixtures/reference-repository/ directory. Before any tests, the testing script renames the git.rename directory to .git, so that spec/fixtures/reference-repository becomes a valid git repository. It undoes that change at the end of tests.
To update the test reference-repository with fresh real datas, remove old repository and copy the new one:
$ rm -rf spec/fixtures/reference-repository
$ cp -r ~/git/reference-repository/data/
As far as I can tell the follwing section on data reduction doesn't work anymore the
git-filter-repo tool doesn't exist anymore (And is probably now part of a standard git command)
and the git gc --aggressive will break the subrepo.
You should only try these steps in a separated commit.
The repository can be quite heavy, so to limit the import size into g5k-api we
can only keep the data directory and rewrite the git history.
The tool git-filter-repo can be used to do that:
$ cd spec/fixtures/reference-repository
$ /usr/libexec/git-core/git-filter-repo --path data
$ git gc --aggressive
** End of broken section **
Some rspec tests require to add a symlink inside the repository, because this aspect is not used inside our reference-repository (we don't use links at all).
For example:
$ rm data/grid5000/sites/nancy/servers/grcinq-srv-3.json
$ ln -s data/grid5000/sites/nancy/servers/grcinq-srv-2.json data/grid5000/sites/nancy/servers/grcinq-srv-3.json
Commit the change, and then move the .git:
$ mv .git git.rename
$ cd ../../..
Update the @latest_commit variable:
$ sed -ri "s/(\@latest_commit = ).*/\1 \'$(cat spec/fixtures/reference-repository/git.rename/refs/heads/master)\'/g" spec/spec_help.rb
Then, changes are to be applied manually, because some resources might not exist anymore, and some might have been added, e.g.:
- the number and name of sites
- the clusters
- nodes inside a clusters
- servers on sites
- network_equipments
- …
However, sed can be usefull to help and save time. Here to replace inside tests
a site (bordeaux) and a cluster (bordemer) which don't exist anymore:
$ find spec/ -type f -exec sed -ri 's/bordeaux/grenoble/g' {} \;
$ find spec/ -type f -exec sed -ri 's/bordemer/dahu/g' {} \;
- Adding data to the test OAR2 database
The test OAR2 database is loaded in the test environment using the RAILS_ENV=test bundle exec rake db:oar:seed task defined in lib/tasks/test.rake. This tasks loads a SQL dump pointed by the SEED environment variable, and defaults to spec/fixtures/oar2_2011-01-07.sql
Updating the OAR2 test db therefore requires either
- Updating the contents of the
spec/fixtures/oar2_2011-01-07.sqldirectly - creating a new dump of the
oar2_testdatabase running on the vagrant box, and updatinglib/tasks/test.raketo use that dump a the new seed (be sure to git add it to the repo). As thatoar2_testdatabase should be a valid oar2 database, its contents should be updatable using oar commands. You'll need to apt-get install oar2 first and configureoar2.confto point to the test database first. If you do so, please consider spending a few minutes to update the puppet provisonning recipes so as to pre-configure the vagrant VM to manage the test database out of the box. Furture developper will thank you for that.
The debian package build is done automatically as a stage in gitlab-ci. See .gitlab-ci.yaml and https://gitlab.inria.fr/grid5000/g5k-api/pipelines , but only tagged commits get pushed to the packages.grid5000.fr repository.
-
Once you've packaged the new version, you must release it to the APT repository hosted on packages.grid5000.fr. Their is a manual step in gitlab's CI for this.
-
If you released on packages.grid5000.fr, then api-server-devel servers will pick it up, but you'll need to change the hiera data for the production versions.
-
Generate general statistics:
$ bundle exec rake statsExample of output you'll get on the STDOUT:
+----------------------+-------+-------+---------+---------+-----+-------+ | Name | Lines | LOC | Classes | Methods | M/C | LOC/M | +----------------------+-------+-------+---------+---------+-----+-------+ | Controllers | 447 | 352 | 14 | 31 | 2 | 9 | | Helpers | 54 | 26 | 0 | 5 | 0 | 3 | | Models | 351 | 288 | 7 | 25 | 3 | 9 | | Libraries | 3 | 3 | 0 | 0 | 0 | 0 | | Model specs | 340 | 315 | 0 | 0 | 0 | 0 | | Controller specs | 550 | 523 | 0 | 0 | 0 | 0 | | Helper specs | 15 | 3 | 0 | 0 | 0 | 0 | +----------------------+-------+-------+---------+---------+-----+-------+ | Total | 1760 | 1510 | 21 | 61 | 2 | 22 | +----------------------+-------+-------+---------+---------+-----+-------+ Code LOC: 669 Test LOC: 841 Code to Test Ratio: 1:1.3 -
Generate the test coverage:
$ RAILS_ENV=test bundle exec rspec spec/ $ open coverage/index.html
- Cyril Rohr [email protected], David Margery [email protected] and others