From 8f485b2d0b38fdf7f67c3c067421924ad4126ef3 Mon Sep 17 00:00:00 2001
From: Syphax bouazzouni <gs_bouazzouni@esi.dz>
Date: Tue, 5 Dec 2023 22:44:08 +0100
Subject: [PATCH 1/6] Merge to master - Release 2.3.6 - add Docker image  (#35)

* add request_lang middleware

* pin redis gem version to 4.8.1

* update Gemfile.lock

* [ontoportal-bot] Gemfile.lock update

* [ontoportal-bot] Gemfile.lock update

* [ontoportal-bot] Gemfile.lock update

* update Gemfile to use ontologies_linked_data new metadata branch

* update TestOntologySubmissionsController to be adapted to the new model

* [ontoportal-bot] Gemfile.lock update

* [ontoportal-bot] Gemfile.lock update

* [ontoportal-bot] Gemfile.lock update

* in submission_metadata rename display with category

* add to submission_metadata description and example fields

* add the option to do pagination for the submission endpoint

* extract retrieve_latest_submissions method to submission helper

* implement apply_filters to submissions endpoint using SPARQL FILTERs

* add test for submissions endpoint pagination

* [ontoportal-bot] Gemfile.lock update

* fix private only submission filter

* add hasFormalityLevel filter for submissions endpoint

* add  for ontology: reviews, notes, projects on the submissions endpoints

* bring submission metrics for submissions  endpoints

* bring all contact attributes if asked in the submissions endpoints

* refactor submissions endpoint filters by extracting some methods

* add ontology acronym or name filters for submissions endpoints

* add submissions endpoint order_by option

* [ontoportal-bot] Gemfile.lock update

* fix including ontology and contacts in the submissions endpoints

* fix list admin filter_access control (e.g for submissions  endpoints)

* [ontoportal-bot] Gemfile.lock update

* check access of ontologies in /ontologies/:acronym/submissions endpoint

* include ontology viewOf attribute in the submission endpoints

* make apply_filters helper generic for any of model attributes

* add Agents controller

* add pagination to agents index endpoint if asked

* make agents routes work for /Agent and /agent

* handle agent indentifiers and affiliations attributes  save and update

* make agent controller work for affiliations attribute

* add agent controller tests

* don't update affiliations if only 'id' sent in params

* bring identifier attributes when we update an agent

* update agent test to work with the new Agent validators

* bring the agent attributes on display all of the submissions endpoints

* handle exception for  class attribute but aren't in populate_from_params

* update Gemfile to use development branch of OLD

* refactor user controller to extract reset password helpers

* remove the send notification on user creation, now handled by user.save

* add access token authentication

* fix test after enforcing the uniqueness of user emails

* fix search test

* add oauth_authentication test

* bring the correct attributes when the oauth_authenticate is used

* Feature: Add support of multilingual search (#40)

* update get_term_search_query to support multilanguages search

* rename var

* fix search lang suffix to use underscore not @

* add multilangual search test

---------

Co-authored-by: Syphax Bouazzouni <gs_bouazzouni@esi.dz>

* add get submission all including all properties test

* extract and use submission_include_params where we use submission.bring

* use retrieve_submissions helper in the :acronym/submissions endpoint

* update Goo version and add submissions filters test

* Fix: display contact for get submissions  (#45)

* add get submission all including all properties test

* extract and use submission_include_params where we use submission.bring

* use retrieve_submissions helper in the :acronym/submissions endpoint

* Fix: Submissions filters with order_by  for the same attribute  (#46)

* add get submission all including all properties test

* extract and use submission_include_params where we use submission.bring

* use retrieve_submissions helper in the :acronym/submissions endpoint

* update Goo version and add submissions filters test

* make the ontology submissions endpoint include views

* include all metrics attribues in the submissions endpoints (#53)

* add ontology submissions filter by status (#56)

* add agent usage attribute tests (#55)

* Fix: optimize fetching all agents usages query by batch loading them (#57)

* add agent usage attribute tests

* optimize fetching all agents usages query by batch loading them

* Feature: Add ontologies_api docker image build CI  (#58)

* add docker build CI

* Feature: add ontoportal bash script (#59)

* add ontoportal bash script to run test and development servers

* update README.md

* update docker CI to work in production releases

* Feature: add ontoportal bash script (#59)

* add ontoportal bash script to run test and development servers

* update README.md

* update docker CI to work in production releases

* fix date list properties population helper

* Feature: update ontoportal bash script to handle local gems binding   (#61)

* add ontoportal bash script to run test and development servers

* update README.md

* update docker CI to work in production releases

* update ontoportal script to handle local gems bindq

* update ontoportal script to handle binding to local gem for development

* fixing the test runner after the new changes in the ontoportal script

* add description filter to the submissions endpoint (#62)

---------

Co-authored-by: OntoPortal Bot <ontoportal.bot.lirmm@gmail.com>
Co-authored-by: HADDAD Zineddine <hz_haddad@esi.dz>
---
 .env.sample                                   |   4 +
 .github/workflows/docker-image.yml            |  55 ++++
 .github/workflows/ruby-unit-tests.yml         |   3 +
 .gitignore                                    |   2 +
 Dockerfile                                    |   3 +-
 Gemfile                                       |   7 +-
 Gemfile.lock                                  |  19 +-
 README.md                                     |  59 ++++-
 bin/ontoportal                                | 239 ++++++++++++++++++
 config/environments/config.rb.sample          | 200 +++++++--------
 controllers/agents_controller.rb              |   5 +
 docker-compose.yml                            |   9 +-
 helpers/application_helper.rb                 |   5 +-
 helpers/request_params_helper.rb              |  29 ++-
 helpers/submission_helper.rb                  |   8 +
 test/controllers/test_agents_controller.rb    |   6 +-
 .../test_ontology_submissions_controller.rb   | 154 ++++++++++-
 17 files changed, 656 insertions(+), 151 deletions(-)
 create mode 100644 .env.sample
 create mode 100644 .github/workflows/docker-image.yml
 create mode 100755 bin/ontoportal

diff --git a/.env.sample b/.env.sample
new file mode 100644
index 00000000..2c15a1c0
--- /dev/null
+++ b/.env.sample
@@ -0,0 +1,4 @@
+API_URL=http://localhost:9393
+ONTOLOGIES_LINKED_DATA_PATH=
+GOO_PATH=
+SPARQL_CLIENT_PATH=
\ No newline at end of file
diff --git a/.github/workflows/docker-image.yml b/.github/workflows/docker-image.yml
new file mode 100644
index 00000000..737482f9
--- /dev/null
+++ b/.github/workflows/docker-image.yml
@@ -0,0 +1,55 @@
+name: Docker branch Images build
+
+on:
+  push:
+    branches:
+      - development
+      - stage
+      - test
+  release:
+    types: [ published ]
+jobs:
+  push_to_registry:
+    name: Push Docker branch image to Docker Hub
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check out the repo
+        uses: actions/checkout@v3
+
+      - name: Set up QEMU
+        uses: docker/setup-qemu-action@v2
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v2
+
+      - name: Log in to Docker Hub
+        uses: docker/login-action@f4ef78c080cd8ba55a85445d5b36e214a81df20a
+        with:
+          username: ${{ secrets.DOCKERHUB_USERNAME }}
+          password: ${{ secrets.DOCKERHUB_TOKEN }}
+
+      - name: Log in to the Container registry
+        uses: docker/login-action@65b78e6e13532edd9afa3aa52ac7964289d1a9c1
+        with:
+          registry: ghcr.io
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Extract metadata (tags, labels) for Docker
+        id: meta
+        uses: docker/metadata-action@v4
+        with:
+          images: |
+            agroportal/ontologies_api
+            ghcr.io/${{ github.repository }}
+
+      - name: Build and push Docker image
+        uses: docker/build-push-action@v4
+        with:
+          context: .
+          platforms: linux/amd64,linux/arm64
+          build-args: |
+            RUBY_VERSION=2.7.8
+          push: true
+          tags: ${{ steps.meta.outputs.tags }}
+          labels: ${{ steps.meta.outputs.labels }}
diff --git a/.github/workflows/ruby-unit-tests.yml b/.github/workflows/ruby-unit-tests.yml
index 6b2c973d..4dc9e323 100644
--- a/.github/workflows/ruby-unit-tests.yml
+++ b/.github/workflows/ruby-unit-tests.yml
@@ -12,6 +12,8 @@ jobs:
     runs-on: ubuntu-latest
     steps:
     - uses: actions/checkout@v3
+    - name: copy-env-config
+      run: cp .env.sample .env
     - name: Build docker-compose
       run: docker-compose --profile 4store build #profile flag is set in order to build all containers in this step
     - name: Run unit tests
@@ -19,6 +21,7 @@ jobs:
       # http://docs.codecov.io/docs/testing-with-docker
       run: |
         ci_env=`bash <(curl -s https://codecov.io/env)`
+        docker-compose run $ci_env -e CI --rm ${{ matrix.backend }} wait-for-it solr-ut:8983 -- bundle install 
         docker-compose run $ci_env -e CI --rm ${{ matrix.backend }} wait-for-it solr-ut:8983 -- bundle exec rake test TESTOPTS='-v'
     - name: Upload coverage reports to Codecov
       uses: codecov/codecov-action@v3
diff --git a/.gitignore b/.gitignore
index 886a220f..8b568832 100644
--- a/.gitignore
+++ b/.gitignore
@@ -71,3 +71,5 @@ test/data/ontology_files/catalog-v001.xml
 create_permissions.log
 
 ontologies_api.iml
+
+.env
diff --git a/Dockerfile b/Dockerfile
index 3e65fe4a..6294e102 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -21,6 +21,7 @@ ENV BUNDLE_PATH=/srv/ontoportal/bundle
 RUN bundle install
 
 COPY . /srv/ontoportal/ontologies_api
+RUN cp /srv/ontoportal/ontologies_api/config/environments/config.rb.sample /srv/ontoportal/ontologies_api/config/environments/development.rb
 
 EXPOSE 9393
-CMD ["bundle", "exec", "rackup", "-p", "9393", "--host", "0.0.0.0"]
+CMD ["bundle", "exec", "rackup", "-p", "9393", "--host", "0.0.0.0"]
\ No newline at end of file
diff --git a/Gemfile b/Gemfile
index 49c8357e..caa9818a 100644
--- a/Gemfile
+++ b/Gemfile
@@ -1,6 +1,6 @@
 source 'https://rubygems.org'
 
-gem 'activesupport', '~> 3.1'
+gem 'activesupport', '~> 3.2'
 # see https://github.com/ncbo/ontologies_api/issues/69
 gem 'bigdecimal', '1.4.2'
 gem 'faraday', '~> 1.9'
@@ -44,12 +44,12 @@ gem 'haml', '~> 5.2.2' # pin see https://github.com/ncbo/ontologies_api/pull/107
 gem 'redcarpet'
 
 # NCBO gems (can be from a local dev path or from rubygems/git)
-gem 'goo', git: 'https://github.com/ontoportal-lirmm/goo.git', branch: 'master'
+gem 'goo', git: 'https://github.com/ontoportal-lirmm/goo.git', branch: 'development'
 gem 'ncbo_annotator', git: 'https://github.com/ontoportal-lirmm/ncbo_annotator.git', branch: 'master'
 gem 'ncbo_cron', git: 'https://github.com/ontoportal-lirmm/ncbo_cron.git', branch: 'master'
 gem 'ncbo_ontology_recommender', git: 'https://github.com/ncbo/ncbo_ontology_recommender.git', branch: 'master'
 gem 'sparql-client', github: 'ontoportal-lirmm/sparql-client', branch: 'master'
-gem 'ontologies_linked_data', git: 'https://github.com/ontoportal-lirmm/ontologies_linked_data.git', branch: 'master'
+gem 'ontologies_linked_data', git: 'https://github.com/ontoportal-lirmm/ontologies_linked_data.git', branch: 'development'
 
 group :development do
   # bcrypt_pbkdf and ed35519 is required for capistrano deployments when using ed25519 keys; see https://github.com/miloserdow/capistrano-deploy/issues/42
@@ -63,6 +63,7 @@ group :development do
   gem 'shotgun', github: 'palexander/shotgun', branch: 'ncbo'
 end
 
+
 group :profiling do
   gem 'rack-mini-profiler'
 end
diff --git a/Gemfile.lock b/Gemfile.lock
index 3428c7b2..52a5c072 100644
--- a/Gemfile.lock
+++ b/Gemfile.lock
@@ -11,8 +11,8 @@ GIT
 
 GIT
   remote: https://github.com/ontoportal-lirmm/goo.git
-  revision: 74ea47defc7f6260b045a6c6997bbe6a59c7bf62
-  branch: master
+  revision: 5979402d5138850fb9bdb34edfa350e9af1b5d22
+  branch: development
   specs:
     goo (0.0.2)
       addressable (~> 2.8)
@@ -53,8 +53,8 @@ GIT
 
 GIT
   remote: https://github.com/ontoportal-lirmm/ontologies_linked_data.git
-  revision: 80a331d053ea04397a903452288c2186822c340c
-  branch: master
+  revision: a199eff007f5d7f18205d61194f3823445aa6460
+  branch: development
   specs:
     ontologies_linked_data (0.0.1)
       activesupport
@@ -216,7 +216,7 @@ GEM
     multi_json (1.15.0)
     multipart-post (2.3.0)
     net-http-persistent (2.9.4)
-    net-imap (0.4.7)
+    net-imap (0.4.6)
       date
       net-protocol
     net-pop (0.1.2)
@@ -247,7 +247,7 @@ GEM
       rack (>= 0.4)
     rack-attack (6.6.1)
       rack (>= 1.0, < 3)
-    rack-cache (1.14.0)
+    rack-cache (1.13.0)
       rack (>= 0.4)
     rack-cors (1.0.6)
       rack (>= 1.6.0)
@@ -270,8 +270,8 @@ GEM
     redis-rack-cache (2.2.1)
       rack-cache (>= 1.10, < 2)
       redis-store (>= 1.6, < 2)
-    redis-store (1.9.2)
-      redis (>= 4, < 6)
+    redis-store (1.9.1)
+      redis (>= 4, < 5)
     representable (3.2.0)
       declarative (< 0.1.0)
       trailblazer-option (>= 0.1.1, < 0.2.0)
@@ -346,12 +346,11 @@ GEM
     webrick (1.8.1)
 
 PLATFORMS
-  x86_64-darwin-21
   x86_64-darwin-23
   x86_64-linux
 
 DEPENDENCIES
-  activesupport (~> 3.1)
+  activesupport (~> 3.2)
   bcrypt_pbkdf (>= 1.0, < 2.0)
   bigdecimal (= 1.4.2)
   capistrano (~> 3)
diff --git a/README.md b/README.md
index dfaa77ea..02b9f076 100644
--- a/README.md
+++ b/README.md
@@ -2,7 +2,50 @@
 
 ontologies_api provides a RESTful interface for accessing [BioPortal](https://bioportal.bioontology.org/) (an open repository of biomedical ontologies). Supported services include downloads, search, access to terms and concepts, text annotation, and much more.
 
-## Prerequisites
+# Run ontologies_api
+
+## Using OntoPortal api utilities script 
+### See help
+
+```bash 
+bin/ontoportal help
+```
+
+```
+Usage: bin/ontoportal {dev|test|run|help} [--reset-cache] [--api-url API_URL] [--api-key API_KEY]
+  dev            : Start the Ontoportal API development server.
+                  Example: bin/ontoportal dev --api-url http://localhost:9393
+                  Use --reset-cache to remove volumes: bin/ontoportal dev --reset-cache
+  test           : Run tests.
+  run            : Run a command in the Ontoportal API Docker container.
+  help           : Show this help message.
+
+Description:
+  This script provides convenient commands for managing an Ontoportal API
+  application using Docker Compose. It includes options for starting the development server,
+  running tests, and executing commands within the Ontoportal API Docker container.
+
+Goals:
+  - Simplify common tasks related to Ontoportal API development using Docker.
+  - Provide a consistent and easy-to-use interface for common actions.
+
+
+```
+
+
+### Run dev
+```bash 
+bin/ontoportal dev 
+```
+
+### Run test with a local OntoPortal API
+```bash 
+bin/ontoportal test 
+```
+
+
+## Manually 
+### Prerequisites
 
 - [Ruby 2.x](http://www.ruby-lang.org/en/downloads/) (most recent patch level)
 - [rbenv](https://github.com/sstephenson/rbenv) and [ruby-build](https://github.com/sstephenson/ruby-build) (optional)
@@ -19,7 +62,7 @@ ontologies_api provides a RESTful interface for accessing [BioPortal](https://bi
 - [Solr](http://lucene.apache.org/solr/)
     - BioPortal indexes ontology class and property content using Solr (a Lucene-based server)
 
-## Configuring Solr
+### Configuring Solr
 
 To configure Solr for ontologies_api usage, modify the example project included with Solr by doing the following:
 
@@ -46,22 +89,22 @@ To configure Solr for ontologies_api usage, modify the example project included
     # Edit the ontologieS_api/config/environments/{env}.rb file to point to your running instance:
     # http://localhost:8983/solr/NCBO1
 
-## Installing
+### Installing
 
-### Clone the repository
+#### Clone the repository
 
 ```
 $ git clone git@github.com:ncbo/ontologies_api.git
 $ cd ontologies_api
 ```
 
-### Install the dependencies
+#### Install the dependencies
 
 ```
 $ bundle install
 ```
 
-### Create an environment configuration file
+#### Create an environment configuration file
 
 ```
 $ cp config/environments/config.rb.sample config/environments/development.rb
@@ -73,7 +116,7 @@ production.rb<br />
 development.rb<br />
 test.rb
 
-### Run the unit tests (optional)
+#### Run the unit tests (optional)
 
 Requires a configuration file for the test environment:
 
@@ -87,7 +130,7 @@ Execute the suite of tests from the command line:
 $ bundle exec rake test 
 ```
 
-### Run the application
+#### Run the application
 
 ```
 $ bundle exec rackup --port 9393 
diff --git a/bin/ontoportal b/bin/ontoportal
new file mode 100755
index 00000000..4840dad3
--- /dev/null
+++ b/bin/ontoportal
@@ -0,0 +1,239 @@
+#!/usr/bin/env bash
+
+# Function to display script usage information
+show_help() {
+  cat << EOL
+Usage: $0 {dev|test|run|help} [--reset-cache] [--api-url API_URL] [--api-key API_KEY] [--old-path OLD_PATH] [--goo-path GOO_PATH] [--sparql-client-path SPARQL_CLIENT_PATH]
+  dev            : Start the Ontoportal API development server.
+                  Example: $0 dev --api-url http://localhost:9393
+                  Use --reset-cache to remove volumes: $0 dev --reset-cache
+  test           : Run tests. Specify either a test file or use 'all'.
+                  Example: $0 test test/controllers/test_users_controller.rb -v --name=name_of_the_test
+                  Example (run all tests): $0 test all -v
+  run            : Run a command in the Ontoportal API Docker container.
+  help           : Show this help message.
+
+Description:
+  This script provides convenient commands for managing an Ontoportal API
+  application using Docker Compose. It includes options for starting the development server,
+  running tests, and executing commands within the Ontoportal API Docker container.
+
+Options:
+  --reset-cache             : Remove Docker volumes (used with 'dev').
+  --api-url API_URL         : Specify the API URL.
+  --api-key API_KEY         : Specify the API key.
+  --old-path OLD_PATH       : Specify the path for ontologies_linked_data.
+  --goo-path GOO_PATH       : Specify the path for goo.
+  --sparql-client-path      : Specify the path for sparql-client.
+  test_file | all            : Specify either a test file or  all the tests will be run.
+  -v                        : Enable verbosity.
+  --name=name_of_the_test   : Specify the name of the test.
+
+Goals:
+  - Simplify common tasks related to Ontoportal API development using Docker.
+  - Provide a consistent and easy-to-use interface for common actions.
+EOL
+}
+
+
+# Function to update or create the .env file with API_URL and API_KEY
+update_env_file() {
+  # Update the .env file with the provided values
+  local api_url="$1"
+  local old_path="$2"
+  local goo_path="$3"
+  local sparql_client_path="$4"
+
+  # Update  the .env file with the provided values
+  file_content=$(<.env)
+
+  # Make changes to the variable
+  while IFS= read -r line; do
+        if [[ "$line" == "API_URL="* && -n "$api_url" ]]; then
+          echo "API_URL=$api_url"
+        elif [[ "$line" == "ONTOLOGIES_LINKED_DATA_PATH="* ]]; then
+          echo "ONTOLOGIES_LINKED_DATA_PATH=$old_path"
+        elif [[ "$line" == "GOO_PATH="* ]]; then
+          echo "GOO_PATH=$goo_path"
+        elif [[ "$line" == "SPARQL_CLIENT_PATH="* ]]; then
+          echo "SPARQL_CLIENT_PATH=$sparql_client_path"
+        else
+          echo "$line"
+        fi
+  done <<< "$file_content" > .env
+}
+
+# Function to create configuration files if they don't exist
+create_config_files() {
+  [ -f ".env" ] || cp .env.sample .env
+  [ -f "config/environments/development.rb" ] || cp config/environments/config.rb.sample config/environments/development.rb
+}
+
+# Function to build Docker run command with conditionally added bind mounts
+build_docker_run_cmd() {
+  local custom_command="$1"
+  local old_path="$2"
+  local goo_path="$3"
+  local sparql_client_path="$4"
+
+  local docker_run_cmd="docker compose run --rm -it"
+  local bash_cmd=""
+
+  # Conditionally add bind mounts only if the paths are not empty
+  for path_var in "old_path:ontologies_linked_data" "goo_path:goo" "sparql_client_path:sparql-client"; do
+    IFS=':' read -r path value <<< "$path_var"
+
+    if [ -n "${!path}" ]; then
+      host_path="$(realpath "$(dirname "${!path}")")/$value"
+      echo "Run: bundle config local.$value ${!path}"
+      container_path="/srv/ontoportal/$value"
+      docker_run_cmd+=" -v $host_path:$container_path"
+      bash_cmd+="(git config --global --add safe.directory $container_path && bundle config local.$value $container_path) &&"
+    else
+      bash_cmd+=" (bundle config unset local.$value) &&"
+    fi
+  done
+
+  bash_cmd+=" (bundle check || bundle install || bundle update) && $custom_command"
+  docker_run_cmd+=" --service-ports api bash -c \"$bash_cmd\""
+
+  eval "$docker_run_cmd"
+}
+
+# Function to handle the "dev" and "test" options
+run_command() {
+  local custom_command="$1"
+
+  local reset_cache=false
+  local api_url=""
+  local old_path=""
+  local goo_path=""
+  local sparql_client_path=""
+
+  shift
+  # Check for command line arguments
+  while [[ "$#" -gt 0 ]]; do
+    case $1 in
+      --reset-cache)
+        reset_cache=true
+        shift
+        ;;
+      --api-url)
+        api_url="$2"
+        shift 2
+        ;;
+      --old-path)
+        old_path="$2"
+        shift 2
+        ;;
+      --goo-path)
+        goo_path="$2"
+        shift 2
+        ;;
+      --sparql-client-path)
+        sparql_client_path="$2"
+        shift 2
+        ;;
+      *)
+        echo "Unknown option: $1"
+        show_help
+        exit 1
+        ;;
+    esac
+  done
+
+  # Check if --reset-cache is present and execute docker compose down --volumes
+  if [ "$reset_cache" = true ]; then
+    echo "Resetting cache. Running: docker compose down --volumes"
+    docker compose down --volumes
+  fi
+
+  # Check if arguments are provided
+  update_env_file "$api_url" "$old_path" "$goo_path" "$sparql_client_path"
+
+
+
+  # If no arguments, fetch values from the .env file
+  source .env
+  api_url="$API_URL"
+  old_path="$ONTOLOGIES_LINKED_DATA_PATH"
+  goo_path="$GOO_PATH"
+  sparql_client_path="$SPARQL_CLIENT_PATH"
+
+
+  if [ -z "$api_url" ] ; then
+    echo "Error: Missing required arguments. Please provide both --api-url or update them in your .env"
+    exit 1
+  fi
+
+
+
+  # Build the Docker run command
+  echo "Run: $custom_command"
+  build_docker_run_cmd "$custom_command" "$old_path" "$goo_path" "$sparql_client_path"
+}
+
+# Function to handle the "dev" option
+dev() {
+  echo "Starting OntoPortal API development server..."
+
+  local custom_command="bundle exec shotgun --host 0.0.0.0  --env=development"
+  run_command "$custom_command" "$@"
+}
+
+# Function to handle the "test" option
+test() {
+  echo "Running tests..."
+  local test_path=""
+  local test_options=""
+  local all_arguments=()
+  # Check for command line arguments
+  while [ "$#" -gt 0 ]; do
+     case "$1" in
+         --api-url | --reset-cache | --old-path | --goo-path | --sparql-client-path)
+          all_arguments+=("$1" "$2")
+          shift 2
+          ;;
+       *)
+         if [ -z "$test_path" ]; then
+           test_path="$1"
+         else
+           test_options="$test_options $1"
+         fi
+         ;;
+     esac
+     shift
+  done
+
+  local custom_command="bundle exec rake test TEST='$test_path' TESTOPTS='$test_options'"
+  echo "run :  $custom_command"
+  run_command "$custom_command" "${all_arguments[@]}"
+}
+
+# Function to handle the "run" option
+run() {
+  echo "Run: $*"
+  docker compose run --rm -it api bash -c "$*"
+}
+
+create_config_files
+
+# Main script logic
+case "$1" in
+  "run")
+    run "${@:2}"
+    ;;
+  "dev")
+    dev "${@:2}"
+    ;;
+  "test")
+    test "${@:2}"
+    ;;
+  "help")
+    show_help
+    ;;
+  *)
+    show_help
+    exit 1
+    ;;
+esac
diff --git a/config/environments/config.rb.sample b/config/environments/config.rb.sample
index e5f9fd9c..8713b9f2 100644
--- a/config/environments/config.rb.sample
+++ b/config/environments/config.rb.sample
@@ -3,120 +3,106 @@
 #   development.rb
 #   test.rb
 
-begin
-  LinkedData.config do |config|
-    config.repository_folder     = "/srv/ncbo/repository"
-    config.goo_host              = "localhost"
-    config.goo_port              = 9000
-    config.search_server_url     = "http://localhost:8082/solr/term_search_core1"
-    config.property_search_server_url = "http://localhost:8082/solr/prop_search_core1"
-    config.rest_url_prefix       = "http://#{$SITE_URL}:8080/"
-    config.replace_url_prefix    = true
-    config.enable_security       = true
-
-    config.apikey                = "24e0e77e-54e0-11e0-9d7b-005056aa3316"
-    config.ui_host               = "http://#{$SITE_URL}"
-    config.enable_monitoring     = false
-    config.cube_host             = "localhost"
-    config.enable_resource_index = false
-
-    # Used to define other BioPortal to which this appliance can be mapped to
-    # Example to map to the NCBO BioPortal : {"ncbo" => {"api" => "http://data.bioontology.org", "ui" => "http://bioportal.bioontology.org", "apikey" => ""}}
-    # Then create the mapping using the following class in JSON : "http://purl.bioontology.org/ontology/MESH/C585345": "ncbo:MESH"
-    # Where "ncbo" is the key in the interportal_hash. Use only lowercase letters for this key.
-    # And do not use "ext" as a key, it is reserved for clases outside of any BioPortal
-    config.interportal_hash   = {}
-
-    # Caches
-    config.http_redis_host    = "localhost"
-    config.http_redis_port    = 6380
-    config.enable_http_cache  = true
-    config.goo_redis_host     = "localhost"
-    config.goo_redis_port     = 6382
+GOO_BACKEND_NAME = ENV.include?("GOO_BACKEND_NAME") ? ENV["GOO_BACKEND_NAME"] : "4store"
+GOO_HOST         = ENV.include?("GOO_HOST")         ? ENV["GOO_HOST"]         : "localhost"
+GOO_PATH_DATA    = ENV.include?("GOO_PATH_DATA")    ? ENV["GOO_PATH_DATA"]    : "/data/"
+GOO_PATH_QUERY   = ENV.include?("GOO_PATH_QUERY")   ? ENV["GOO_PATH_QUERY"]   : "/sparql/"
+GOO_PATH_UPDATE  = ENV.include?("GOO_PATH_UPDATE")  ? ENV["GOO_PATH_UPDATE"]  : "/update/"
+GOO_PORT         = ENV.include?("GOO_PORT")         ? ENV["GOO_PORT"]         : 9000
+MGREP_HOST       = ENV.include?("MGREP_HOST")       ? ENV["MGREP_HOST"]       : "localhost"
+MGREP_PORT       = ENV.include?("MGREP_PORT")       ? ENV["MGREP_PORT"]       : 55555
+MGREP_DICTIONARY_FILE = ENV.include?("MGREP_DICTIONARY_FILE") ? ENV["MGREP_DICTIONARY_FILE"] : "./test/data/dictionary.txt"
+REDIS_GOO_CACHE_HOST  = ENV.include?("REDIS_GOO_CACHE_HOST")  ? ENV["REDIS_GOO_CACHE_HOST"]  : "localhost"
+REDIS_HTTP_CACHE_HOST = ENV.include?("REDIS_HTTP_CACHE_HOST") ? ENV["REDIS_HTTP_CACHE_HOST"] : "localhost"
+REDIS_PERSISTENT_HOST = ENV.include?("REDIS_PERSISTENT_HOST") ? ENV["REDIS_PERSISTENT_HOST"] : "localhost"
+REDIS_PORT            = ENV.include?("REDIS_PORT")            ? ENV["REDIS_PORT"]            : 6379
+REPORT_PATH           = ENV.include?("REPORT_PATH")           ? ENV["REPORT_PATH"]           : "./test/ontologies_report.json"
+REPOSITORY_FOLDER     = ENV.include?("REPOSITORY_FOLDER")     ? ENV["REPOSITORY_FOLDER"]     : "./test/data/ontology_files/repo"
+REST_URL_PREFIX       = ENV.include?("REST_URL_PREFIX")       ? ENV["REST_URL_PREFIX"]       : ENV["API_URL"] || "http://localhost:9393"
+SOLR_PROP_SEARCH_URL  = ENV.include?("SOLR_PROP_SEARCH_URL")  ? ENV["SOLR_PROP_SEARCH_URL"]  : "http://localhost:8983/solr/prop_search_core1"
+SOLR_TERM_SEARCH_URL  = ENV.include?("SOLR_TERM_SEARCH_URL")  ? ENV["SOLR_TERM_SEARCH_URL"]  : "http://localhost:8983/solr/term_search_core1"
 
-    Goo.use_cache             = true
-
-    # Email notifications
-    config.enable_notifications   = false
-    config.email_sender           = "admin@example.org" # Default sender for emails
-    config.email_override         = "override@example.org" # all email gets sent here. Disable with email_override_disable.
-    config.email_disable_override = true
-    config.smtp_host              = "localhost"
-    config.smtp_port              = 25
-    config.smtp_auth_type         = :none # :none, :plain, :login, :cram_md5
-    config.smtp_domain            = "example.org"
-    # Emails of the instance administrators to get mail notifications when new user or new ontology
-    config.admin_emails           = ["admin@example.org"]
+begin
+  # For prefLabel extract main_lang first, or anything if no main found.
+  # For other properties only properties with a lang that is included in main_lang are used
+  Goo.main_languages = ["en", "fr"]
+  Goo.use_cache = false
+rescue NoMethodError
+  puts "(CNFG) >> Goo.main_lang not available"
+end
 
-    # PURL server config parameters
-    config.enable_purl            = false
-    config.purl_host              = "purl.example.org"
-    config.purl_port              = 80
-    config.purl_username          = "admin"
-    config.purl_password          = "password"
-    config.purl_maintainers       = "admin"
-    config.purl_target_url_prefix = "http://example.org"
+LinkedData.config do |config|
+  config.goo_backend_name              = GOO_BACKEND_NAME.to_s
+  config.goo_host                      = GOO_HOST.to_s
+  config.goo_port                      = GOO_PORT.to_i
+  config.goo_path_query                = GOO_PATH_QUERY.to_s
+  config.goo_path_data                 = GOO_PATH_DATA.to_s
+  config.goo_path_update               = GOO_PATH_UPDATE.to_s
+  config.goo_redis_host                = REDIS_GOO_CACHE_HOST.to_s
+  config.goo_redis_port                = REDIS_PORT.to_i
+  config.http_redis_host               = REDIS_HTTP_CACHE_HOST.to_s
+  config.http_redis_port               = REDIS_PORT.to_i
+  config.ontology_analytics_redis_host = REDIS_PERSISTENT_HOST.to_s
+  config.ontology_analytics_redis_port = REDIS_PORT.to_i
+  config.search_server_url             = SOLR_TERM_SEARCH_URL.to_s
+  config.property_search_server_url    = SOLR_PROP_SEARCH_URL.to_s
+  config.replace_url_prefix            = true
+  config.rest_url_prefix               = REST_URL_PREFIX.to_s
+#  config.enable_notifications          = false
 
-    # Ontology Google Analytics Redis
-    # disabled
-    config.ontology_analytics_redis_host = "localhost"
-    config.enable_ontology_analytics = false
-    config.ontology_analytics_redis_port = 6379
-  end
-rescue NameError
-  puts "(CNFG) >> LinkedData not available, cannot load config"
+    config.interportal_hash = {
+      "agroportal" => {
+        "api" => "http://data.agroportal.lirmm.fr",
+        "ui" => "http://agroportal.lirmm.fr",
+        "apikey" => "1cfae05f-9e67-486f-820b-b393dec5764b"
+      },
+      "ncbo" => {
+        "api" => "http://data.bioontology.org",
+        "apikey" => "4a5011ea-75fa-4be6-8e89-f45c8c84844e",
+        "ui" => "http://bioportal.bioontology.org",
+      },
+      "sifr" => {
+        "api" => "http://data.bioportal.lirmm.fr",
+        "ui" => "http://bioportal.lirmm.fr",
+        "apikey" => "1cfae05f-9e67-486f-820b-b393dec5764b"
+      }
+    }
+    config.oauth_providers = {
+      github: {
+        check: :access_token,
+        link: 'https://api.github.com/user'
+      },
+      keycloak: {
+        check: :jwt_token,
+        cert: 'KEYCLOAK_SECRET_KEY'
+      },
+      orcid: {
+        check: :access_token,
+        link: 'https://pub.orcid.org/v3.0/me'
+      },
+      google: {
+        check: :access_token,
+        link: 'https://www.googleapis.com/oauth2/v3/userinfo'
+      }
+    }
 end
 
-begin
-  Annotator.config do |config|
-    config.mgrep_dictionary_file   = "/srv/mgrep/dictionary/dictionary.txt"
-    config.stop_words_default_file = "./config/default_stop_words.txt"
-    config.mgrep_host              = "localhost"
-    config.mgrep_port              = 55555
-    config.mgrep_alt_host          = "localhost"
-    config.mgrep_alt_port          = 55555
-    config.annotator_redis_host    = "localhost"
-    config.annotator_redis_port    = 6379
-  end
-rescue NameError
-  puts "(CNFG) >> Annotator not available, cannot load config"
+Annotator.config do |config|
+  config.annotator_redis_host  = REDIS_PERSISTENT_HOST.to_s
+  config.annotator_redis_port  = REDIS_PORT.to_i
+  config.mgrep_host            = MGREP_HOST.to_s
+  config.mgrep_port            = MGREP_PORT.to_i
+  config.mgrep_dictionary_file = MGREP_DICTIONARY_FILE.to_s
 end
 
 LinkedData::OntologiesAPI.config do |config|
-  config.restrict_download = ["ACR0", "ACR1", "ACR2"]
-end
-
-begin
-  LinkedData::OntologiesAPI.config do |config|
-    config.enable_unicorn_workerkiller = true
-    config.enable_throttling           = false
-    config.enable_monitoring           = false
-    config.cube_host                   = "localhost"
-    config.http_redis_host             = "localhost"
-    config.http_redis_port             = 6380
-    config.ontology_rank               = ""
-    config.resolver_redis_host = "localhost"
-    config.resolver_redis_port = 6379
-    config.restrict_download = ["ACR0", "ACR1", "ACR2"]
-  end
-rescue NameError
-  puts "(CNFG) >> OntologiesAPI not available, cannot load config"
+  config.http_redis_host = REDIS_HTTP_CACHE_HOST.to_s
+  config.http_redis_port = REDIS_PORT.to_i
+#  config.restrict_download = ["ACR0", "ACR1", "ACR2"]
 end
 
-begin
-  NcboCron.config do |config|
-    config.redis_host                = Annotator.settings.annotator_redis_host
-    config.redis_port                = Annotator.settings.annotator_redis_port
-    config.enable_ontology_analytics = false
-    config.enable_ontologies_report  = false
-    # Schedulues
-    config.cron_schedule             = "30 */4 * * *"
-    # Pull schedule
-    config.pull_schedule             = "00 18 * * *"
-    # Pull long schedule for ontology that are pulled less frequently: run weekly on monday at 11 a.m. (23:00)
-    config.pull_schedule_long                       = "00 23 * * 1"
-    config.pull_long_ontologies                     = ["BIOREFINERY", "TRANSMAT", "GO"]
-  end
-rescue NameError
-  puts "(CNFG) >> NcboCron not available, cannot load config"
-end
+NcboCron.config do |config|
+  config.redis_host = REDIS_PERSISTENT_HOST.to_s
+  config.redis_port = REDIS_PORT.to_i
+  config.ontology_report_path = REPORT_PATH
+end
\ No newline at end of file
diff --git a/controllers/agents_controller.rb b/controllers/agents_controller.rb
index 87572e99..1bf86321 100644
--- a/controllers/agents_controller.rb
+++ b/controllers/agents_controller.rb
@@ -14,6 +14,11 @@ class AgentsController < ApplicationController
         else
           agents = query.to_a
         end
+
+        if includes_param.include?(:all) || includes_param.include?(:usages)
+          LinkedData::Models::Agent.load_agents_usages(agents)
+        end
+
         reply agents
       end
 
diff --git a/docker-compose.yml b/docker-compose.yml
index 5cb64963..f7325381 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -1,10 +1,5 @@
 x-app: &app
-    build:
-      context: .
-      args:
-        RUBY_VERSION: '2.7'
-    # Increase the version number in the image tag every time Dockerfile or its arguments is changed
-    image: ontologies_api:0.0.1
+    image: agroportal/ontologies_api:development
     environment: &env
       BUNDLE_PATH: /srv/ontoportal/bundle
       # default bundle config resolves to /usr/local/bundle/config inside of the container
@@ -39,6 +34,8 @@ x-app: &app
 services:
   api:
     <<: *app
+    env_file:
+      .env
     environment:
       <<: *env
       GOO_BACKEND_NAME: 4store
diff --git a/helpers/application_helper.rb b/helpers/application_helper.rb
index 172170fa..d90630a3 100644
--- a/helpers/application_helper.rb
+++ b/helpers/application_helper.rb
@@ -88,7 +88,10 @@ def populate_from_params(obj, params)
             value = retrieved_values
           elsif attribute_settings && attribute_settings[:enforce] && attribute_settings[:enforce].include?(:date_time)
             # TODO: Remove this awful hack when obj.class.model_settings[:range][attribute] contains DateTime class
-            value = DateTime.parse(value)
+            is_array = value.is_a?(Array)
+            value = Array(value).map{ |v| DateTime.parse(v) }
+            value = value.first unless is_array
+            value
           elsif attribute_settings && attribute_settings[:enforce] && attribute_settings[:enforce].include?(:uri) && attribute_settings[:enforce].include?(:list)
             # in case its a list of URI, convert all value to IRI
             value = value.map { |v| RDF::IRI.new(v) }
diff --git a/helpers/request_params_helper.rb b/helpers/request_params_helper.rb
index 842ee0a7..59adeba7 100644
--- a/helpers/request_params_helper.rb
+++ b/helpers/request_params_helper.rb
@@ -45,9 +45,9 @@ def apply_submission_filters(query)
           isOfType: params[:isOfType]&.split(','), #["http://omv.ontoware.org/2005/05/ontology#Vocabulary"],
           hasFormalityLevel: params[:hasFormalityLevel]&.split(','), #["http://w3id.org/nkos/nkostype#thesaurus"],
           ontology_viewingRestriction: params[:viewingRestriction]&.split(','), #["private"]
+          status: params[:status]&.split(','), #"retired",
         }
         inverse_filters = {
-          status: params[:status], #"retired",
           submissionStatus: params[:submissionStatus] #"RDF",
         }
 
@@ -122,17 +122,24 @@ def add_inverse_filters(inverse_filters, query)
       end
 
       def add_acronym_name_filters(query)
-        if params[:acronym]
-          filter = Goo::Filter.new(extract_attr(:ontology_acronym)).regex(params[:acronym])
-          if params[:name]
-            filter.or(Goo::Filter.new(extract_attr(:ontology_name)).regex(params[:name]))
-          end
-          query = query.filter(filter)
-        elsif params[:name]
-          filter = Goo::Filter.new(extract_attr(:ontology_name)).regex(params[:name])
-          query = query.filter(filter)
+        filters = {
+          acronym: :ontology_acronym,
+          name: :ontology_name,
+          description: :description
+        }.map do |key, attr|
+          (params[key].nil? || params[key].empty?) ?  nil : [extract_attr(attr), params[key]]
+        end.compact
+
+        return  query if filters.empty?
+
+        key, val = filters.first
+        filter = Goo::Filter.new(key).regex(val)
+
+        filters.drop(1).each do |k, v|
+          filter = filter.or(Goo::Filter.new(k).regex(v))
         end
-        query
+        
+        query.filter(filter)
       end
 
       def add_order_by_patterns(query)
diff --git a/helpers/submission_helper.rb b/helpers/submission_helper.rb
index 07f82138..b79737d0 100644
--- a/helpers/submission_helper.rb
+++ b/helpers/submission_helper.rb
@@ -13,6 +13,14 @@ def submission_include_params
         if includes.find{|v| v.is_a?(Hash) && v.keys.include?(:contact)}
           includes << {:contact=>[:name, :email]}
         end
+
+        if includes.find{|v| v.is_a?(Hash) && v.keys.include?(:metrics)}
+          includes << { metrics: [:maxChildCount, :properties, :classesWithMoreThan25Children,
+                                  :classesWithOneChild, :individuals, :maxDepth, :classes,
+                                  :classesWithNoDefinition, :averageChildCount, :numberOfAxioms,
+                                  :entities]}
+        end
+
         includes
       end
 
diff --git a/test/controllers/test_agents_controller.rb b/test/controllers/test_agents_controller.rb
index ef0e5c47..de36bc36 100644
--- a/test/controllers/test_agents_controller.rb
+++ b/test/controllers/test_agents_controller.rb
@@ -28,14 +28,14 @@ def teardown
   end
 
   def test_all_agents
-    get '/agents'
+    get '/agents?display=all&page=1'
     assert last_response.ok?
 
     created_agents = MultiJson.load(last_response.body)
-
     @agents.each do |agent|
-      created_agent = created_agents.select{|x| x["name"].eql?(agent[:name])}.first
+      created_agent = created_agents["collection"].select{|x| x["name"].eql?(agent[:name])}.first
       refute_nil created_agent
+      refute_nil created_agent["usages"]
       assert_equal agent[:name], created_agent["name"]
       assert_equal agent[:identifiers].size, created_agent["identifiers"].size
       assert_equal agent[:identifiers].map{|x| x[:notation]}.sort, created_agent["identifiers"].map{|x| x['notation']}.sort
diff --git a/test/controllers/test_ontology_submissions_controller.rb b/test/controllers/test_ontology_submissions_controller.rb
index 77b6e6bc..095d0339 100644
--- a/test/controllers/test_ontology_submissions_controller.rb
+++ b/test/controllers/test_ontology_submissions_controller.rb
@@ -202,7 +202,7 @@ def test_download_acl_only
   end
 
   def test_submissions_pagination
-    num_onts_created, created_ont_acronyms = create_ontologies_and_submissions(ont_count: 2, submission_count: 2)
+    num_onts_created, created_ont_acronyms, ontologies = create_ontologies_and_submissions(ont_count: 2, submission_count: 2)
 
     get "/submissions"
     assert last_response.ok?
@@ -217,6 +217,158 @@ def test_submissions_pagination
     assert_equal 1, submissions["collection"].length
   end
 
+  def test_submissions_pagination_filter
+    num_onts_created, created_ont_acronyms, ontologies = create_ontologies_and_submissions(ont_count: 10, submission_count: 1)
+    group1 = LinkedData::Models::Group.new(acronym: 'group-1', name: "Test Group 1").save
+    group2 = LinkedData::Models::Group.new(acronym: 'group-2', name: "Test Group 2").save
+    category1 = LinkedData::Models::Category.new(acronym: 'category-1', name: "Test Category 1").save
+    category2 = LinkedData::Models::Category.new(acronym: 'category-2', name: "Test Category 2").save
+
+    ontologies1 = ontologies[0..5].each do |o|
+      o.bring_remaining
+      o.group = [group1]
+      o.hasDomain = [category1]
+      o.save
+    end
+
+    ontologies2 = ontologies[6..8].each do |o|
+      o.bring_remaining
+      o.group = [group2]
+      o.hasDomain = [category2]
+      o.save
+    end
+
+
+
+    # test filter by group and category
+    get "/submissions?page=1&pagesize=100&group=#{group1.acronym}"
+    assert last_response.ok?
+    assert_equal ontologies1.size, MultiJson.load(last_response.body)["collection"].length
+    get "/submissions?page=1&pagesize=100&group=#{group2.acronym}"
+    assert last_response.ok?
+    assert_equal ontologies2.size, MultiJson.load(last_response.body)["collection"].length
+    get "/submissions?page=1&pagesize=100&hasDomain=#{category1.acronym}"
+    assert last_response.ok?
+    assert_equal ontologies1.size, MultiJson.load(last_response.body)["collection"].length
+    get "/submissions?page=1&pagesize=100&hasDomain=#{category2.acronym}"
+    assert last_response.ok?
+    assert_equal ontologies2.size, MultiJson.load(last_response.body)["collection"].length
+    get "/submissions?page=1&pagesize=100&hasDomain=#{category2.acronym}&group=#{group1.acronym}"
+    assert last_response.ok?
+    assert_equal 0, MultiJson.load(last_response.body)["collection"].length
+    get "/submissions?page=1&pagesize=100&hasDomain=#{category2.acronym}&group=#{group2.acronym}"
+    assert last_response.ok?
+    assert_equal ontologies2.size, MultiJson.load(last_response.body)["collection"].length
+
+    ontologies3 = ontologies[9]
+    ontologies3.bring_remaining
+    ontologies3.group = [group1, group2]
+    ontologies3.hasDomain = [category1, category2]
+    ontologies3.name = "name search test"
+    ontologies3.save
+
+    # test search with acronym
+    [
+      [ 1, ontologies.first.acronym],
+      [ 1, ontologies.last.acronym],
+      [ontologies.size, 'TEST-ONT']
+    ].each do |count, acronym_search|
+      get "/submissions?page=1&pagesize=100&acronym=#{acronym_search}"
+      assert last_response.ok?
+      submissions = MultiJson.load(last_response.body)
+      assert_equal count, submissions["collection"].length
+    end
+
+
+    # test search with name
+    [
+      [ 1, ontologies.first.name],
+      [ 1, ontologies.last.name],
+      [ontologies.size - 1, 'TEST-ONT']
+    ].each do |count, name_search|
+      get "/submissions?page=1&pagesize=100&name=#{name_search}"
+      assert last_response.ok?
+      submissions = MultiJson.load(last_response.body)
+      binding.pry unless submissions["collection"].length.eql?(count)
+      assert_equal count, submissions["collection"].length
+    end
+
+    # test search with name and acronym
+    # search by name
+    get "/submissions?page=1&pagesize=100&name=search&acronym=search"
+    assert last_response.ok?
+    submissions = MultiJson.load(last_response.body)
+    assert_equal 1, submissions["collection"].length
+    # search by acronym
+    get "/submissions?page=1&pagesize=100&name=9&acronym=9"
+    assert last_response.ok?
+    submissions = MultiJson.load(last_response.body)
+    assert_equal 1, submissions["collection"].length
+    # search by acronym or name
+    get "/submissions?page=1&pagesize=100&name=search&acronym=8"
+    assert last_response.ok?
+    submissions = MultiJson.load(last_response.body)
+    assert_equal 2, submissions["collection"].length
+
+    ontologies.first.name = "sort by test"
+    ontologies.first.save
+    sub = ontologies.first.latest_submission(status: :any).bring_remaining
+    sub.status = 'retired'
+    sub.description = "234"
+    sub.creationDate = DateTime.yesterday.to_datetime
+    sub.hasOntologyLanguage = LinkedData::Models::OntologyFormat.find('SKOS').first
+    sub.save
+
+    #test search with sort
+    get "/submissions?page=1&pagesize=100&acronym=tes&name=tes&order_by=ontology_name"
+    assert last_response.ok?
+    submissions = MultiJson.load(last_response.body)
+    refute_empty submissions["collection"]
+    assert_equal ontologies.map{|x| x.name}.sort, submissions["collection"].map{|x| x["ontology"]["name"]}
+
+    get "/submissions?page=1&pagesize=100&acronym=tes&name=tes&order_by=creationDate"
+    assert last_response.ok?
+    submissions = MultiJson.load(last_response.body)
+    refute_empty submissions["collection"]
+    assert_equal ontologies.map{|x| x.latest_submission(status: :any).bring(:creationDate).creationDate}.sort.map(&:to_s), submissions["collection"].map{|x| x["creationDate"]}.reverse
+
+    # test search with format
+    get "/submissions?page=1&pagesize=100&acronym=tes&name=tes&hasOntologyLanguage=SKOS"
+    assert last_response.ok?
+    submissions = MultiJson.load(last_response.body)
+    refute_empty submissions["collection"]
+    assert_equal 1, submissions["collection"].size
+
+    get "/submissions?page=1&pagesize=100&acronym=tes&name=tes&hasOntologyLanguage=OWL"
+    assert last_response.ok?
+    submissions = MultiJson.load(last_response.body)
+    refute_empty submissions["collection"]
+    assert_equal ontologies.size-1 , submissions["collection"].size
+
+    # test ontology filter with submission filter attributes
+    get "/submissions?page=1&pagesize=100&acronym=tes&name=tes&group=group-2&category=category-2&hasOntologyLanguage=OWL"
+    assert last_response.ok?
+    submissions = MultiJson.load(last_response.body)
+    refute_empty submissions["collection"]
+    assert_equal ontologies2.size + 1 , submissions["collection"].size
+
+    # test ontology filter with status
+    get "/submissions?page=1&pagesize=100&status=retired"
+    assert last_response.ok?
+    submissions = MultiJson.load(last_response.body)
+    refute_empty submissions["collection"]
+    assert_equal 1 , submissions["collection"].size
+
+    get "/submissions?page=1&pagesize=100&status=alpha,beta,production"
+    assert last_response.ok?
+    submissions = MultiJson.load(last_response.body)
+    refute_empty submissions["collection"]
+    assert_equal ontologies.size - 1 , submissions["collection"].size
+    get "/submissions?page=1&pagesize=100&description=234&acronym=234&name=234"
+    assert last_response.ok?
+    submissions = MultiJson.load(last_response.body)
+    assert_equal 1 , submissions["collection"].size
+  end
 
   def test_submissions_default_includes
     ontology_count = 5

From 7d32a7024f30117fbeeccc1d4a6fe3ecd2be798f Mon Sep 17 00:00:00 2001
From: OntoPortal Bot <ontoportal.bot.lirmm@gmail.com>
Date: Tue, 12 Dec 2023 16:51:05 +0100
Subject: [PATCH 2/6] [ontoportal-bot] Gemfile.lock update

---
 Gemfile.lock | 24 +++++++++++++-----------
 1 file changed, 13 insertions(+), 11 deletions(-)

diff --git a/Gemfile.lock b/Gemfile.lock
index 52a5c072..c6454cc8 100644
--- a/Gemfile.lock
+++ b/Gemfile.lock
@@ -11,7 +11,7 @@ GIT
 
 GIT
   remote: https://github.com/ontoportal-lirmm/goo.git
-  revision: 5979402d5138850fb9bdb34edfa350e9af1b5d22
+  revision: 9aa0ccacc9d76bff096218e6d48edc5b0bffd54f
   branch: development
   specs:
     goo (0.0.2)
@@ -37,7 +37,7 @@ GIT
 
 GIT
   remote: https://github.com/ontoportal-lirmm/ncbo_cron.git
-  revision: 8db3481116c57d2a21dc8f32bcd1695d95442280
+  revision: d424a0335ac1e7334cf97e5aa5b7f5b94414c9d8
   branch: master
   specs:
     ncbo_cron (0.0.1)
@@ -53,7 +53,7 @@ GIT
 
 GIT
   remote: https://github.com/ontoportal-lirmm/ontologies_linked_data.git
-  revision: a199eff007f5d7f18205d61194f3823445aa6460
+  revision: 1cfaf4482b7bf8c9001f0ff309f2a0ff06f683b5
   branch: development
   specs:
     ontologies_linked_data (0.0.1)
@@ -103,7 +103,7 @@ GEM
     activesupport (3.2.22.5)
       i18n (~> 0.6, >= 0.6.4)
       multi_json (~> 1.0)
-    addressable (2.8.5)
+    addressable (2.8.6)
       public_suffix (>= 2.0.2, < 6.0)
     airbrussh (1.5.0)
       sshkit (>= 1.6.1, != 1.7.0)
@@ -173,8 +173,11 @@ GEM
       retriable (>= 2.0, < 4.a)
       rexml
       webrick
-    googleauth (1.8.1)
-      faraday (>= 0.17.3, < 3.a)
+    google-cloud-env (2.0.1)
+      faraday (>= 1.0, < 3.a)
+    googleauth (1.9.0)
+      faraday (>= 1.0, < 3.a)
+      google-cloud-env (~> 2.0, >= 2.0.1)
       jwt (>= 1.4, < 3.0)
       multi_json (~> 1.11)
       os (>= 0.9, < 2.0)
@@ -207,7 +210,7 @@ GEM
     method_source (1.0.0)
     mime-types (3.5.1)
       mime-types-data (~> 3.2015)
-    mime-types-data (3.2023.1003)
+    mime-types-data (3.2023.1205)
     mini_mime (1.1.5)
     minitest (4.7.5)
     minitest-stub_any_instance (1.0.3)
@@ -216,7 +219,7 @@ GEM
     multi_json (1.15.0)
     multipart-post (2.3.0)
     net-http-persistent (2.9.4)
-    net-imap (0.4.6)
+    net-imap (0.4.8)
       date
       net-protocol
     net-pop (0.1.2)
@@ -251,7 +254,7 @@ GEM
       rack (>= 0.4)
     rack-cors (1.0.6)
       rack (>= 1.6.0)
-    rack-mini-profiler (3.1.1)
+    rack-mini-profiler (3.3.0)
       rack (>= 1.2.0)
     rack-protection (1.5.5)
       rack
@@ -346,7 +349,6 @@ GEM
     webrick (1.8.1)
 
 PLATFORMS
-  x86_64-darwin-23
   x86_64-linux
 
 DEPENDENCIES
@@ -403,4 +405,4 @@ DEPENDENCIES
   webmock
 
 BUNDLED WITH
-   2.4.21
+   2.3.23

From b941c21fdaac31a1168981a9f2109b54357b1ef5 Mon Sep 17 00:00:00 2001
From: Syphax bouazzouni <gs_bouazzouni@esi.dz>
Date: Sat, 16 Dec 2023 23:37:14 +0100
Subject: [PATCH 3/6] update API documentation to the language parameter  to
 search

---
 Gemfile                                |  4 ++--
 Gemfile.lock                           | 19 ++++++++++---------
 views/documentation/documentation.haml |  1 +
 3 files changed, 13 insertions(+), 11 deletions(-)

diff --git a/Gemfile b/Gemfile
index caa9818a..e2a713f1 100644
--- a/Gemfile
+++ b/Gemfile
@@ -44,12 +44,12 @@ gem 'haml', '~> 5.2.2' # pin see https://github.com/ncbo/ontologies_api/pull/107
 gem 'redcarpet'
 
 # NCBO gems (can be from a local dev path or from rubygems/git)
-gem 'goo', git: 'https://github.com/ontoportal-lirmm/goo.git', branch: 'development'
+gem 'goo', git: 'https://github.com/ontoportal-lirmm/goo.git', branch: 'master'
 gem 'ncbo_annotator', git: 'https://github.com/ontoportal-lirmm/ncbo_annotator.git', branch: 'master'
 gem 'ncbo_cron', git: 'https://github.com/ontoportal-lirmm/ncbo_cron.git', branch: 'master'
 gem 'ncbo_ontology_recommender', git: 'https://github.com/ncbo/ncbo_ontology_recommender.git', branch: 'master'
 gem 'sparql-client', github: 'ontoportal-lirmm/sparql-client', branch: 'master'
-gem 'ontologies_linked_data', git: 'https://github.com/ontoportal-lirmm/ontologies_linked_data.git', branch: 'development'
+gem 'ontologies_linked_data', git: 'https://github.com/ontoportal-lirmm/ontologies_linked_data.git', branch: 'master'
 
 group :development do
   # bcrypt_pbkdf and ed35519 is required for capistrano deployments when using ed25519 keys; see https://github.com/miloserdow/capistrano-deploy/issues/42
diff --git a/Gemfile.lock b/Gemfile.lock
index c6454cc8..cdac8983 100644
--- a/Gemfile.lock
+++ b/Gemfile.lock
@@ -11,8 +11,8 @@ GIT
 
 GIT
   remote: https://github.com/ontoportal-lirmm/goo.git
-  revision: 9aa0ccacc9d76bff096218e6d48edc5b0bffd54f
-  branch: development
+  revision: 03da25b671d2ffa515b5dce51c6bd35980ae60c7
+  branch: master
   specs:
     goo (0.0.2)
       addressable (~> 2.8)
@@ -53,8 +53,8 @@ GIT
 
 GIT
   remote: https://github.com/ontoportal-lirmm/ontologies_linked_data.git
-  revision: 1cfaf4482b7bf8c9001f0ff309f2a0ff06f683b5
-  branch: development
+  revision: 2878f43e71cfb83b86dbd54a86587a1a635bafb8
+  branch: master
   specs:
     ontologies_linked_data (0.0.1)
       activesupport
@@ -173,11 +173,11 @@ GEM
       retriable (>= 2.0, < 4.a)
       rexml
       webrick
-    google-cloud-env (2.0.1)
+    google-cloud-env (2.1.0)
       faraday (>= 1.0, < 3.a)
-    googleauth (1.9.0)
+    googleauth (1.9.1)
       faraday (>= 1.0, < 3.a)
-      google-cloud-env (~> 2.0, >= 2.0.1)
+      google-cloud-env (~> 2.1)
       jwt (>= 1.4, < 3.0)
       multi_json (~> 1.11)
       os (>= 0.9, < 2.0)
@@ -185,7 +185,7 @@ GEM
     haml (5.2.2)
       temple (>= 0.8.0)
       tilt
-    hashdiff (1.0.1)
+    hashdiff (1.1.0)
     http-accept (1.7.0)
     http-cookie (1.0.5)
       domain_name (~> 0.5)
@@ -349,6 +349,7 @@ GEM
     webrick (1.8.1)
 
 PLATFORMS
+  x86_64-darwin-23
   x86_64-linux
 
 DEPENDENCIES
@@ -405,4 +406,4 @@ DEPENDENCIES
   webmock
 
 BUNDLED WITH
-   2.3.23
+   2.4.21
diff --git a/views/documentation/documentation.haml b/views/documentation/documentation.haml
index 527d781f..916eb1e7 100644
--- a/views/documentation/documentation.haml
+++ b/views/documentation/documentation.haml
@@ -151,6 +151,7 @@
               %li include={prefLabel, synonym, definition, notation, cui, semanticType} // default = (see Common Parameters section)
               %li page={integer representing the page number} // default = 1
               %li pagesize={integer representing the size of the returned page} // default = 50
+              %li language={an ISO 639-1 language value, e.g 'fr' or 'en'} // by default search in all languages
 
         %h4#nav_search_subtree Subtree Search
 

From a05edfc90d5e89fcb2078081f69f521bf55da667 Mon Sep 17 00:00:00 2001
From: OntoPortal Bot <ontoportal.bot.lirmm@gmail.com>
Date: Sat, 16 Dec 2023 23:42:45 +0100
Subject: [PATCH 4/6] [ontoportal-bot] Gemfile.lock update

---
 Gemfile.lock | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/Gemfile.lock b/Gemfile.lock
index cdac8983..d85d6911 100644
--- a/Gemfile.lock
+++ b/Gemfile.lock
@@ -349,7 +349,6 @@ GEM
     webrick (1.8.1)
 
 PLATFORMS
-  x86_64-darwin-23
   x86_64-linux
 
 DEPENDENCIES
@@ -406,4 +405,4 @@ DEPENDENCIES
   webmock
 
 BUNDLED WITH
-   2.4.21
+   2.3.23

From 165723bc5a8ecaa2cf2aa3055c96d796016747af Mon Sep 17 00:00:00 2001
From: OntoPortal Bot <ontoportal.bot.lirmm@gmail.com>
Date: Wed, 20 Dec 2023 06:41:38 +0100
Subject: [PATCH 5/6] [ontoportal-bot] Gemfile.lock update

---
 Gemfile.lock | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/Gemfile.lock b/Gemfile.lock
index d85d6911..0ed977a0 100644
--- a/Gemfile.lock
+++ b/Gemfile.lock
@@ -37,7 +37,7 @@ GIT
 
 GIT
   remote: https://github.com/ontoportal-lirmm/ncbo_cron.git
-  revision: d424a0335ac1e7334cf97e5aa5b7f5b94414c9d8
+  revision: 155db7a33794f03858893d2367cb119f27726a31
   branch: master
   specs:
     ncbo_cron (0.0.1)
@@ -53,7 +53,7 @@ GIT
 
 GIT
   remote: https://github.com/ontoportal-lirmm/ontologies_linked_data.git
-  revision: 2878f43e71cfb83b86dbd54a86587a1a635bafb8
+  revision: e98b884999e5ce917a8be5fdc37f7b4797a1559e
   branch: master
   specs:
     ontologies_linked_data (0.0.1)
@@ -230,7 +230,7 @@ GEM
       net-ssh (>= 2.6.5, < 8.0.0)
     net-smtp (0.4.0)
       net-protocol
-    net-ssh (7.2.0)
+    net-ssh (7.2.1)
     netrc (0.11.0)
     newrelic_rpm (9.6.0)
       base64

From 422cbb548dd7eeb3bdf5818e7a2e006e78b02875 Mon Sep 17 00:00:00 2001
From: Syphax bouazzouni <gs_bouazzouni@esi.dz>
Date: Thu, 28 Dec 2023 21:03:39 +0100
Subject: [PATCH 6/6] add analytics controller with  endpoints for users,
 ontologies and pages (#63)

---
 Gemfile.lock                                  |  4 +-
 controllers/analytics_controller.rb           | 45 +++++++++++++++++++
 helpers/application_helper.rb                 |  4 +-
 test/controllers/test_annotator_controller.rb |  6 +--
 test/controllers/test_mappings_controller.rb  |  2 +-
 .../test_ontology_analytics_controller.rb     |  4 +-
 6 files changed, 56 insertions(+), 9 deletions(-)
 create mode 100644 controllers/analytics_controller.rb

diff --git a/Gemfile.lock b/Gemfile.lock
index 52a5c072..579efc0e 100644
--- a/Gemfile.lock
+++ b/Gemfile.lock
@@ -53,7 +53,7 @@ GIT
 
 GIT
   remote: https://github.com/ontoportal-lirmm/ontologies_linked_data.git
-  revision: a199eff007f5d7f18205d61194f3823445aa6460
+  revision: d5af89ba5563cbd9bfc06c51a8330015dd737614
   branch: development
   specs:
     ontologies_linked_data (0.0.1)
@@ -216,7 +216,7 @@ GEM
     multi_json (1.15.0)
     multipart-post (2.3.0)
     net-http-persistent (2.9.4)
-    net-imap (0.4.6)
+    net-imap (0.4.9)
       date
       net-protocol
     net-pop (0.1.2)
diff --git a/controllers/analytics_controller.rb b/controllers/analytics_controller.rb
new file mode 100644
index 00000000..1fe6d561
--- /dev/null
+++ b/controllers/analytics_controller.rb
@@ -0,0 +1,45 @@
+require 'csv'
+
+class OntologyAnalyticsController < ApplicationController
+
+  ##
+  # get all ontology analytics for a given year/month combination
+  # TODO use a namespace analytics after migration the old OntologyAnalyticsController
+  namespace "/data/analytics" do
+
+    get 'ontologies' do
+      expires 86400, :public
+      year = year_param(params)
+      error 400, "The year you supplied is invalid. Valid years start with 2 and contain 4 digits." if params["year"] && !year
+      month = month_param(params)
+      error 400, "The month you supplied is invalid. Valid months are 1-12." if params["month"] && !month
+      acronyms = restricted_ontologies_to_acronyms(params)
+      analytics = Ontology.analytics(year, month, acronyms)
+
+      reply analytics
+    end
+
+
+    get 'users' do
+      expires 86400, :public
+      year = year_param(params)
+      error 400, "The year you supplied is invalid. Valid years start with 2 and contain 4 digits." if params["year"] && !year
+      month = month_param(params)
+      error 400, "The month you supplied is invalid. Valid months are 1-12." if params["month"] && !month
+      analytics = User.analytics(year, month)
+      reply analytics['all_users']
+    end
+
+    get 'page_visits' do
+      expires 86400, :public
+      year = year_param(params)
+      error 400, "The year you supplied is invalid. Valid years start with 2 and contain 4 digits." if params["year"] && !year
+      month = month_param(params)
+      error 400, "The month you supplied is invalid. Valid months are 1-12." if params["month"] && !month
+      analytics = User.page_visits_analytics
+      reply analytics['all_pages']
+    end
+
+  end
+
+end
diff --git a/helpers/application_helper.rb b/helpers/application_helper.rb
index d90630a3..24893eef 100644
--- a/helpers/application_helper.rb
+++ b/helpers/application_helper.rb
@@ -276,7 +276,7 @@ def month_param(params=nil)
         if params["month"]
           month = params["month"].strip
           if %r{(?<month>^(0[1-9]|[1-9]|1[0-2])$)}x === month
-            month.to_i
+            month.to_i.to_s
           end
         end
       end
@@ -287,7 +287,7 @@ def year_param(params=nil)
         if params["year"]
           year = params["year"].strip
           if %r{(?<year>^([1-2]\d{3})$)}x === year
-            year.to_i
+            year.to_i.to_s
           end
         end
       end
diff --git a/test/controllers/test_annotator_controller.rb b/test/controllers/test_annotator_controller.rb
index ffa65a97..47f45f40 100644
--- a/test/controllers/test_annotator_controller.rb
+++ b/test/controllers/test_annotator_controller.rb
@@ -260,16 +260,16 @@ def test_default_properties_output
     assert last_response.ok?
     annotations = MultiJson.load(last_response.body)
     assert_equal 9, annotations.length
-    annotations.sort! { |a,b| a["annotatedClass"]["prefLabel"].downcase <=> b["annotatedClass"]["prefLabel"].downcase }
+    annotations.sort! { |a,b| a["annotatedClass"]["prefLabel"].first.downcase <=> b["annotatedClass"]["prefLabel"].first.downcase }
     assert_equal "http://bioontology.org/ontologies/BiomedicalResourceOntology.owl#Aggregate_Human_Data", annotations.first["annotatedClass"]["@id"]
-    assert_equal "Aggregate Human Data", annotations.first["annotatedClass"]["prefLabel"]
+    assert_equal "Aggregate Human Data", Array(annotations.first["annotatedClass"]["prefLabel"]).first
 
     params = {text: text, include: "prefLabel,definition"}
     get "/annotator", params
     assert last_response.ok?
     annotations = MultiJson.load(last_response.body)
     assert_equal 9, annotations.length
-    annotations.sort! { |a,b| a["annotatedClass"]["prefLabel"].downcase <=> b["annotatedClass"]["prefLabel"].downcase }
+    annotations.sort! { |a,b| Array(a["annotatedClass"]["prefLabel"]).first.downcase <=> Array(b["annotatedClass"]["prefLabel"]).first.downcase }
     assert_equal "http://bioontology.org/ontologies/BiomedicalResourceOntology.owl#Aggregate_Human_Data", annotations.first["annotatedClass"]["@id"]
     assert_equal ["A resource that provides data from clinical care that comprises combined data from multiple individual human subjects."], annotations.first["annotatedClass"]["definition"]
   end
diff --git a/test/controllers/test_mappings_controller.rb b/test/controllers/test_mappings_controller.rb
index 52c3975d..cff52225 100644
--- a/test/controllers/test_mappings_controller.rb
+++ b/test/controllers/test_mappings_controller.rb
@@ -245,7 +245,7 @@ def mappings_with_display
     get "/ontologies/#{ontology}/mappings?pagesize=#{pagesize}&page=#{page}&display=prefLabel"
     assert last_response.ok?
     mappings = MultiJson.load(last_response.body)
-    assert mappings["collection"].all? { |m| m["classes"].all? { |c| c["prefLabel"].is_a?(String) && c["prefLabel"].length > 0 } }
+    assert mappings["collection"].all? { |m| m["classes"].all? { |c| c["prefLabel"].first.is_a?(String) && c["prefLabel"].first.length > 0 } }
 
     def_count = 0
     next_page = 1
diff --git a/test/controllers/test_ontology_analytics_controller.rb b/test/controllers/test_ontology_analytics_controller.rb
index 67ab5529..7f2df926 100644
--- a/test/controllers/test_ontology_analytics_controller.rb
+++ b/test/controllers/test_ontology_analytics_controller.rb
@@ -203,7 +203,9 @@ def self.before_suite
       puts "   This test cannot be run because there #{db_size} redis entries (max #{MAX_TEST_REDIS_SIZE}). You are probably pointing to the wrong redis backend. "
       return
     end
-    @@redis.set(LinkedData::Models::Ontology::ONTOLOGY_ANALYTICS_REDIS_FIELD, Marshal.dump(ANALYTICS_DATA))
+
+    stringy_keys = ANALYTICS_DATA.transform_values{|year| year.map{|k,v| [k.to_s , v.stringify_keys]}.to_h}
+    @@redis.set(LinkedData::Models::Ontology::ONTOLOGY_ANALYTICS_REDIS_FIELD, Marshal.dump(stringy_keys))
     @@onts = {
         "NCIT" => "NCIT Ontology",
         "ONTOMA" => "ONTOMA Ontology",