Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix CanarieAPI config to support Python 3.x #284

Merged
merged 53 commits into from
Apr 20, 2023
Merged

Conversation

fmigneault
Copy link
Collaborator

@fmigneault fmigneault commented Feb 1, 2023

Overview

Please include a summary of the changes and which issues are fixed.

Please also include relevant motivation and context.

List any dependencies that are required for this change.

Breaking changes

  • CanarieAPI: update to 0.7.1.

    • The Docker running CanarieAPI is now using Python 3 (since 0.4.x tags).
      Configurations need to be updated if any specific Python 2 definitions were used.
      See 2to3 to help migrate configurations automatically if necessary.
    • Update the CanarieAPI configuration to use
      Python 3.x executable code.

Changes

  • CanarieAPI: update to 0.7.1.

    • The server node now provides a generic server configuration for the current platform definition.
    • Added multiple missing docuementation references for all the services included within CanarieAPI configurations.
    • With new CanarieAPI version, a slightly improved UI with more service details are provided for the active server:

image

  • Add optional variables witht defaults to define reference Docker image version tags.

    Following optional variables are defined by default. These are used as reference in the respective Docker compose
    service definition of these components, as well as in their CanarieAPI configuration to retrieve the release time
    of the tag, and refer to relevant URL references as needed.

    • CATALOG_VERSION
    • FINCH_VERSION
    • FLYINGPIGEON_VERSION
    • GEOSERVER_VERSION
    • HUMMINGBIRD_VERSION
    • MALLEEFOWL_VERSION
    • RAVEN_VERSION

Fixes:

  • CanarieAPI: update to 0.7.1.

    • Fixes an AttributeError raised due to misconfiguration of the Web Application with Flask 2.x definitions
      (relates to Ouranosinc/CanarieAPI#10).
    • Skip over 0.4.x, 0.5.x, 0.6.x versions to avoid issue related to cron job monitoring and log parser
      command failures in order to collect configured service statistics and statuses
      (see also Ouranosinc/CanarieAPI#14).
  • Weaver: update CanarieAPI monitoring definitions

    • Move monitoring of public endpoint under [optional-components/canarie-api-full-monitoring][canarie-monitor].
    • Add monitoring of private endpoint by default when using Weaver component.
  • Cowbird: update CanarieAPI monitoring definitions

    • Add monitoring of public endpoint under [optional-components/canarie-api-full-monitoring][canarie-monitor].
    • Add public Magpie permission on Cowbird entrypoint only to allow its monitoring.

Additional Information

Resolves the following log error.

proxy              | [2023-01-31 19:37:01 +0000] [37] [DEBUG] GET /canarie/
proxy              | [2023-01-31 19:37:01,708] [37] [INFO] app_object : Disconnecting from database
proxy              | [2023-01-31 19:37:01,709] [37] [DEBUG] app_object : Using db filename : /opt/local/src/CanarieAPI/stats.db
proxy              | [2023-01-31 19:37:01 +0000] [37] [DEBUG] Closing connection. 
proxy              | [2023-01-31 19:37:02 +0000] [37] [DEBUG] GET /canarie/background.jpg
proxy              | [2023-01-31 19:37:02,176] [37] [INFO] app_object : Disconnecting from database
proxy              | [2023-01-31 19:37:02,176] [37] [DEBUG] app_object : Using db filename : /opt/local/src/CanarieAPI/stats.db
proxy              | [2023-01-31 19:37:02 +0000] [37] [ERROR] Error handling request /canarie/background.jpg
proxy              | Traceback (most recent call last):
proxy              |   File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/async.py", line 56, in handle
proxy              |     self.handle_request(listener_name, req, client, addr)
proxy              |   File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/ggevent.py", line 152, in handle_request
proxy              |     super(GeventWorker, self).handle_request(*args)
proxy              |   File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/async.py", line 107, in handle_request
proxy              |     respiter = self.wsgi(environ, resp.start_response)
proxy              |   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1997, in __call__
proxy              |     return self.wsgi_app(environ, start_response)
proxy              |   File "/opt/local/src/CanarieAPI/canarieapi/reverse_proxied.py", line 33, in __call__
proxy              |     return self.app(environ, start_response)
proxy              |   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1985, in wsgi_app
proxy              |     response = self.handle_exception(e)
proxy              |   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1532, in handle_exception
proxy              |     handler = self._find_error_handler(InternalServerError())
proxy              |   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1449, in _find_error_handler
proxy              |     .get(code))
proxy              |   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1440, in find_handler
proxy              |     handler = handler_map.get(cls)
proxy              | AttributeError: 'function' object has no attribute 'get'

The above problem lead to unresponsive CanarieAPI, which in turn caused the platform to fail responding with successful monitoring statuses for requests toward the configured services, which in turn, caused the stack to never completely boot. When the stack failed to boot, the End2End Test Results (example: #283 (comment)) could not run due to the unresponsive instance, which is the cause of the incomplete output:

Tests URL :

NOTEBOOK TEST RESULTS
   
````
</code>
``

To Do (in follow-up PRs)

@fmigneault fmigneault requested review from tlvu and matprov February 1, 2023 00:03
@fmigneault fmigneault self-assigned this Feb 1, 2023
@crim-jenkins-bot
Copy link
Collaborator

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac/54/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : fix-canarieapi
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-88.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL :

NOTEBOOK TEST RESULTS
    
</code>

@crim-jenkins-bot
Copy link
Collaborator

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/1159/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : fix-canarieapi
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-154.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL :

NOTEBOOK TEST RESULTS
    
</code>

@fmigneault
Copy link
Collaborator Author

run tests

using required https://github.com/Ouranosinc/CanarieAPI/releases/tag/0.4.4 now deployed

@crim-jenkins-bot
Copy link
Collaborator

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac/55/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : fix-canarieapi
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-154.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL :

NOTEBOOK TEST RESULTS
    
</code>

@crim-jenkins-bot
Copy link
Collaborator

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/1160/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : fix-canarieapi
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-154.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL :

NOTEBOOK TEST RESULTS
    
</code>

Copy link
Collaborator

@tlvu tlvu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The above problem lead to unresponsive CanarieAPI, which in turn caused the platform to fail responding with successful monitoring statuses for requests toward the configured services, which in turn, caused the stack to never completely boot.

It's funny that the error happens during the E2E test. I don't remember seeing this error on our production instance.

Missing CHANGES.md update, small changes to revert and the PR description needs a little review I think.

@crim-jenkins-bot
Copy link
Collaborator

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/1161/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : fix-canarieapi
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : false
PAVICS_HOST : https://host-140-154.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL :

NOTEBOOK TEST RESULTS
    
</code>

@tlvu
Copy link
Collaborator

tlvu commented Feb 1, 2023

Wait, this PR is supposed to fix the E2E? In the latest E2E result, the result still empty!?

@fmigneault
Copy link
Collaborator Author

@tlvu While testing, I identified another issue where CanarieAPI is monitored in Jenkins to check that the stack is "ready", but the value it looks for does not exist. There's a PR open that @matprov needs to review and integrate beforehand. Should be fixed soon enough 🤞

@fmigneault
Copy link
Collaborator Author

run tests

@matprov
Copy link
Collaborator

matprov commented Feb 2, 2023

@fmigneault Jenkins was restarting while running latest E2E tests, therefore I trigger them again.

run tests

@crim-jenkins-bot
Copy link
Collaborator

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac/58/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : fix-canarieapi
DACCS_CONFIGS_BRANCH :
PAVICS_E2E_WORKFLOW_TESTS_BRANCH :
PAVICS_SDI_BRANCH :

DESTROY_INFRA_ON_EXIT :
PAVICS_HOST : https://

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL :

NOTEBOOK TEST RESULTS
    
</code>

@crim-jenkins-bot
Copy link
Collaborator

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/1164/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : fix-canarieapi
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-88.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL :

NOTEBOOK TEST RESULTS
    
</code>

@matprov
Copy link
Collaborator

matprov commented Feb 2, 2023

run tests

@crim-jenkins-bot
Copy link
Collaborator

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/1165/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : fix-canarieapi
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-88.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL :

NOTEBOOK TEST RESULTS
    
</code>

@crim-jenkins-bot
Copy link
Collaborator

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/1175/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : fix-canarieapi
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-46.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL :

NOTEBOOK TEST RESULTS
    
</code>

@crim-jenkins-bot
Copy link
Collaborator

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/1176/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : fix-canarieapi
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-46.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL :

NOTEBOOK TEST RESULTS
    
</code>

@fmigneault
Copy link
Collaborator Author

fmigneault commented Feb 3, 2023

Working on a new version to provide more capabilities: Ouranosinc/CanarieAPI#11

With changes in 76743a4, I get the following result on a local instance :
image

@crim-jenkins-bot
Copy link
Collaborator

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/1177/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : fix-canarieapi
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-46.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL :

NOTEBOOK TEST RESULTS
    
</code>

@crim-jenkins-bot
Copy link
Collaborator

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/1193/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : fix-canarieapi
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL :

NOTEBOOK TEST RESULTS
    
</code>

@bird-house bird-house deleted a comment from crim-jenkins-bot Apr 19, 2023
@bird-house bird-house deleted a comment from crim-jenkins-bot Apr 19, 2023
@crim-jenkins-bot
Copy link
Collaborator

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/1494/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : fix-canarieapi
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-20.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1086/

NOTEBOOK TEST RESULTS
    
[2023-04-19T17:33:22.835Z] ============================= test session starts ==============================
[2023-04-19T17:33:22.835Z] platform linux -- Python 3.8.16, pytest-7.2.0, pluggy-1.0.0
[2023-04-19T17:33:22.835Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2023-04-19T17:33:22.835Z] plugins: anyio-3.6.2, dash-2.7.0, nbval-0.9.6, tornasync-0.6.0.post2
[2023-04-19T17:33:22.835Z] collected 268 items
[2023-04-19T17:33:22.835Z] 
[2023-04-19T17:33:32.280Z] notebooks-auth/test_thredds.ipynb ...........                            [  4%]
[2023-04-19T17:33:43.468Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [  6%]
[2023-04-19T17:33:56.135Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [  8%]
[2023-04-19T17:34:06.088Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 11%]
[2023-04-19T17:34:08.759Z] pavics-sdi-master/docs/source/notebooks/WPS_example.ipynb ..........     [ 15%]
[2023-04-19T17:45:03.996Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 20%]
[2023-04-19T17:45:06.549Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 20%]
[2023-04-19T17:45:16.593Z] ...............                                                          [ 26%]
[2023-04-19T17:45:27.235Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 27%]
[2023-04-19T17:45:38.397Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 30%]
[2023-04-19T17:45:59.208Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 32%]
[2023-04-19T17:46:01.152Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 32%]
[2023-04-19T17:46:14.381Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 35%]
[2023-04-19T17:46:19.672Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb ....F       [ 37%]
[2023-04-19T17:50:09.587Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 42%]
[2023-04-19T17:51:25.534Z] .............                                                            [ 47%]
[2023-04-19T17:51:27.983Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 49%]
[2023-04-19T17:51:30.575Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 52%]
[2023-04-19T17:52:00.545Z] .................                                                        [ 58%]
[2023-04-19T17:52:08.886Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb .....           [ 60%]
[2023-04-19T17:52:10.802Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 60%]
[2023-04-19T17:52:11.850Z] .FFFFFFFF                                                                [ 64%]
[2023-04-19T17:52:24.473Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 68%]
[2023-04-19T17:52:34.474Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 70%]
[2023-04-19T17:53:17.151Z] finch-master/docs/source/notebooks/subset.ipynb ....................ss.  [ 79%]
[2023-04-19T17:53:19.705Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 79%]
[2023-04-19T17:53:31.638Z] ......                                                                   [ 81%]
[2023-04-19T17:57:38.254Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 82%]
[2023-04-19T17:58:57.644Z] .............                                                            [ 86%]
[2023-04-19T17:59:15.788Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 87%]
[2023-04-19T18:04:07.257Z] ....s.                                                                   [ 89%]
[2023-04-19T18:04:15.417Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 89%]
[2023-04-19T18:04:21.247Z] ...                                                                      [ 91%]
[2023-04-19T18:04:36.161Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 91%]
[2023-04-19T18:05:02.353Z] ......                                                                   [ 93%]
[2023-04-19T18:05:03.448Z] notebooks/hummingbird.ipynb ............                                 [ 98%]
[2023-04-19T18:08:38.927Z] notebooks/stress-tests.ipynb .....                                       [100%]
[2023-04-19T18:08:38.927Z] 
[2023-04-19T18:08:38.927Z] =================================== FAILURES ===================================
    
  

@tlvu
Copy link
Collaborator

tlvu commented Apr 19, 2023

canarie-api: failed retrieving version [0.5_dev] release time from [github: bird-house/hummingbird]                                                                                                          
Traceback (most recent call last):                                                                                                                                                                           
  File "/config/docker_configuration.py", line 83, in get_release_time_from_repo_tag                                                                                                                         
    release = _get_release_time_from_github_tag(repository_reference, tagged_version)                                                                                                                        
  File "/config/docker_configuration.py", line 54, in _get_release_time_from_github_tag                                                                                                                      
    raise ValueError(f"No matching version in tagged releases under [{url}].")                                                                                                                               
ValueError: No matching version in tagged releases under [https://api.github.com/repos/bird-house/hummingbird/tags]. 

@tlvu
Copy link
Collaborator

tlvu commented Apr 19, 2023

https://lvupavicsmaster.ouranos.ca/canarie/node/service/status, the lastStatusUpdate do not change. The /stats page is fine.

@fmigneault
Copy link
Collaborator Author

@tlvu
f4da986 resolves the hummingbird tag

Not sure about the lastStatusUpdate though. Any errors in logs regarding a failing operation?

@fmigneault
Copy link
Collaborator Author

run tests

@bird-house bird-house deleted a comment from crim-jenkins-bot Apr 19, 2023
@tlvu
Copy link
Collaborator

tlvu commented Apr 19, 2023

Not sure about the lastStatusUpdate though. Any errors in logs regarding a failing operation?

Nothing in the logs, I think bug in the DB update somewhere, not in the monitoring config in this birdhouse-deploy repo.

@crim-jenkins-bot
Copy link
Collaborator

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/1498/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : fix-canarieapi
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-216.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1087/

NOTEBOOK TEST RESULTS
    
[2023-04-19T23:09:31.073Z] ============================= test session starts ==============================
[2023-04-19T23:09:31.073Z] platform linux -- Python 3.8.16, pytest-7.2.0, pluggy-1.0.0
[2023-04-19T23:09:31.073Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2023-04-19T23:09:31.073Z] plugins: anyio-3.6.2, dash-2.7.0, nbval-0.9.6, tornasync-0.6.0.post2
[2023-04-19T23:09:31.073Z] collected 268 items
[2023-04-19T23:09:31.073Z] 
[2023-04-19T23:09:40.957Z] notebooks-auth/test_thredds.ipynb ...........                            [  4%]
[2023-04-19T23:09:52.802Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [  6%]
[2023-04-19T23:10:01.966Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [  8%]
[2023-04-19T23:10:11.898Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 11%]
[2023-04-19T23:10:16.568Z] pavics-sdi-master/docs/source/notebooks/WPS_example.ipynb ..........     [ 15%]
[2023-04-19T23:23:02.442Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 20%]
[2023-04-19T23:23:02.443Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 20%]
[2023-04-19T23:23:07.845Z] ...............                                                          [ 26%]
[2023-04-19T23:23:18.868Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 27%]
[2023-04-19T23:23:27.568Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 30%]
[2023-04-19T23:23:49.616Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 32%]
[2023-04-19T23:23:51.267Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 32%]
[2023-04-19T23:23:56.723Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 35%]
[2023-04-19T23:24:03.223Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 37%]
[2023-04-19T23:28:21.447Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 42%]
[2023-04-19T23:29:39.013Z] .............                                                            [ 47%]
[2023-04-19T23:29:41.564Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 49%]
[2023-04-19T23:29:44.166Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 52%]
[2023-04-19T23:30:23.425Z] .................                                                        [ 58%]
[2023-04-19T23:30:30.195Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb .....           [ 60%]
[2023-04-19T23:30:32.106Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 60%]
[2023-04-19T23:30:33.155Z] .FFFFFFFF                                                                [ 64%]
[2023-04-19T23:30:46.329Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 68%]
[2023-04-19T23:30:56.813Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 70%]
[2023-04-19T23:31:46.619Z] finch-master/docs/source/notebooks/subset.ipynb ....................ss.  [ 79%]
[2023-04-19T23:31:49.167Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 79%]
[2023-04-19T23:32:06.646Z] ......                                                                   [ 81%]
[2023-04-19T23:37:43.337Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 82%]
[2023-04-19T23:39:40.614Z] .............                                                            [ 86%]
[2023-04-19T23:40:02.598Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 87%]
[2023-04-19T23:48:02.690Z] ....s.                                                                   [ 89%]
[2023-04-19T23:48:10.817Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 89%]
[2023-04-19T23:48:17.199Z] ...                                                                      [ 91%]
[2023-04-19T23:48:32.100Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 91%]
[2023-04-19T23:48:59.386Z] ......                                                                   [ 93%]
[2023-04-19T23:49:01.427Z] notebooks/hummingbird.ipynb ............                                 [ 98%]
[2023-04-19T23:51:32.004Z] notebooks/stress-tests.ipynb ...F.                                       [100%]
[2023-04-19T23:51:32.004Z] 
[2023-04-19T23:51:32.004Z] =================================== FAILURES ===================================
    
  

@tlvu
Copy link
Collaborator

tlvu commented Apr 20, 2023

canarie-api: failed retrieving version [0.5_dev] release time from [github: bird-house/hummingbird]                                                                                                          
Traceback (most recent call last):                                                                                                                                                                           
  File "/config/docker_configuration.py", line 83, in get_release_time_from_repo_tag                                                                                                                         
    release = _get_release_time_from_github_tag(repository_reference, tagged_version)                                                                                                                        
  File "/config/docker_configuration.py", line 54, in _get_release_time_from_github_tag                                                                                                                      
    raise ValueError(f"No matching version in tagged releases under [{url}].")                                                                                                                               
ValueError: No matching version in tagged releases under [https://api.github.com/repos/bird-house/hummingbird/tags]. 

Confirm fixed

canarie-api: found version [0.5_dev] release time [2020-01-21T20:20:21.953117Z] from [docker: pavics/hummingbird] 

@crim-jenkins-bot
Copy link
Collaborator

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/1507/
Result : success

BIRDHOUSE_DEPLOY_BRANCH : fix-canarieapi
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-36.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1093/

NOTEBOOK TEST RESULTS
    
[2023-04-20T16:33:32.606Z] ============================= test session starts ==============================
[2023-04-20T16:33:32.606Z] platform linux -- Python 3.8.16, pytest-7.2.0, pluggy-1.0.0
[2023-04-20T16:33:32.606Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2023-04-20T16:33:32.606Z] plugins: anyio-3.6.2, dash-2.7.0, nbval-0.9.6, tornasync-0.6.0.post2
[2023-04-20T16:33:32.606Z] collected 268 items
[2023-04-20T16:33:32.606Z] 
[2023-04-20T16:33:41.305Z] notebooks-auth/test_thredds.ipynb ...........                            [  4%]
[2023-04-20T16:33:52.646Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [  6%]
[2023-04-20T16:34:01.424Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [  8%]
[2023-04-20T16:34:10.132Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 11%]
[2023-04-20T16:34:12.456Z] pavics-sdi-master/docs/source/notebooks/WPS_example.ipynb ..........     [ 15%]
[2023-04-20T16:41:45.049Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 20%]
[2023-04-20T16:41:47.606Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 20%]
[2023-04-20T16:41:54.231Z] ...............                                                          [ 26%]
[2023-04-20T16:42:04.701Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 27%]
[2023-04-20T16:42:12.622Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 30%]
[2023-04-20T16:42:28.985Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 32%]
[2023-04-20T16:42:30.373Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 32%]
[2023-04-20T16:42:35.307Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 35%]
[2023-04-20T16:42:41.182Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 37%]
[2023-04-20T16:46:12.872Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 42%]
[2023-04-20T16:47:25.794Z] .............                                                            [ 47%]
[2023-04-20T16:47:29.431Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 49%]
[2023-04-20T16:47:32.027Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 52%]
[2023-04-20T16:47:55.217Z] .................                                                        [ 58%]
[2023-04-20T16:48:01.785Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb .....           [ 60%]
[2023-04-20T16:48:03.168Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 60%]
[2023-04-20T16:48:19.787Z] .........                                                                [ 64%]
[2023-04-20T16:48:29.854Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 68%]
[2023-04-20T16:48:39.839Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 70%]
[2023-04-20T16:49:10.410Z] finch-master/docs/source/notebooks/subset.ipynb ....................ss.  [ 79%]
[2023-04-20T16:49:11.796Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 79%]
[2023-04-20T16:49:15.622Z] ......                                                                   [ 81%]
[2023-04-20T16:49:42.210Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 82%]
[2023-04-20T16:50:04.050Z] .............                                                            [ 86%]
[2023-04-20T16:50:14.055Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 87%]
[2023-04-20T16:51:18.785Z] ....s.                                                                   [ 89%]
[2023-04-20T16:51:26.921Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 89%]
[2023-04-20T16:51:32.404Z] ...                                                                      [ 91%]
[2023-04-20T16:51:47.330Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 91%]
[2023-04-20T16:52:13.492Z] ......                                                                   [ 93%]
[2023-04-20T16:52:15.114Z] notebooks/hummingbird.ipynb ............                                 [ 98%]
[2023-04-20T16:55:49.277Z] notebooks/stress-tests.ipynb .....                                       [100%]
[2023-04-20T16:55:49.277Z] 
[2023-04-20T16:55:49.277Z] ================= 265 passed, 3 skipped in 1332.06s (0:22:12) ==================
    
  

@fmigneault fmigneault requested a review from tlvu April 20, 2023 17:28
Copy link
Collaborator

@tlvu tlvu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good enough for now. Please open an issue to not forget to address the issue about performing an install in the entrypoint of canarie-api which defeat the reproducibility feature of a docker image.

@tlvu
Copy link
Collaborator

tlvu commented Apr 20, 2023

Forget to say, this one needs a minor bump instead of a patch bump because of all the image format change.

@github-actions github-actions bot added ci/operations Continuous Integration components documentation Improvements or additions to documentation labels Apr 20, 2023
@fmigneault fmigneault merged commit cf54b20 into master Apr 20, 2023
@fmigneault fmigneault deleted the fix-canarieapi branch April 20, 2023 20:46
@crim-jenkins-bot
Copy link
Collaborator

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/1508/
Result : success

BIRDHOUSE_DEPLOY_BRANCH : fix-canarieapi
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-36.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1094/

NOTEBOOK TEST RESULTS
    
[2023-04-20T20:59:11.628Z] ============================= test session starts ==============================
[2023-04-20T20:59:11.628Z] platform linux -- Python 3.8.16, pytest-7.2.0, pluggy-1.0.0
[2023-04-20T20:59:11.628Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2023-04-20T20:59:11.628Z] plugins: anyio-3.6.2, dash-2.7.0, nbval-0.9.6, tornasync-0.6.0.post2
[2023-04-20T20:59:11.628Z] collected 268 items
[2023-04-20T20:59:11.628Z] 
[2023-04-20T20:59:20.794Z] notebooks-auth/test_thredds.ipynb ...........                            [  4%]
[2023-04-20T20:59:29.093Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [  6%]
[2023-04-20T20:59:36.446Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [  8%]
[2023-04-20T20:59:45.469Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 11%]
[2023-04-20T20:59:48.331Z] pavics-sdi-master/docs/source/notebooks/WPS_example.ipynb ..........     [ 15%]
[2023-04-20T21:07:33.486Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 20%]
[2023-04-20T21:07:36.039Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 20%]
[2023-04-20T21:07:58.446Z] ...............                                                          [ 26%]
[2023-04-20T21:08:13.236Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 27%]
[2023-04-20T21:08:21.153Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 30%]
[2023-04-20T21:08:37.244Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 32%]
[2023-04-20T21:08:38.631Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 32%]
[2023-04-20T21:08:44.027Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 35%]
[2023-04-20T21:08:51.284Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 37%]
[2023-04-20T21:12:18.812Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 42%]
[2023-04-20T21:13:26.011Z] .............                                                            [ 47%]
[2023-04-20T21:13:30.471Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 49%]
[2023-04-20T21:13:32.806Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 52%]
[2023-04-20T21:13:56.214Z] .................                                                        [ 58%]
[2023-04-20T21:14:02.225Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb .....           [ 60%]
[2023-04-20T21:14:03.613Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 60%]
[2023-04-20T21:14:13.549Z] .........                                                                [ 64%]
[2023-04-20T21:14:26.340Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 68%]
[2023-04-20T21:14:36.613Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 70%]
[2023-04-20T21:15:08.716Z] finch-master/docs/source/notebooks/subset.ipynb ....................ss.  [ 79%]
[2023-04-20T21:15:10.635Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 79%]
[2023-04-20T21:15:14.787Z] ......                                                                   [ 81%]
[2023-04-20T21:16:11.075Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 82%]
[2023-04-20T21:16:34.612Z] .............                                                            [ 86%]
[2023-04-20T21:16:46.863Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 87%]
[2023-04-20T21:17:54.618Z] ....s.                                                                   [ 89%]
[2023-04-20T21:18:01.219Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 89%]
[2023-04-20T21:18:07.056Z] ...                                                                      [ 91%]
[2023-04-20T21:18:21.966Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 91%]
[2023-04-20T21:18:45.694Z] ......                                                                   [ 93%]
[2023-04-20T21:18:48.973Z] notebooks/hummingbird.ipynb ............                                 [ 98%]
[2023-04-20T21:22:38.238Z] notebooks/stress-tests.ipynb .....                                       [100%]
[2023-04-20T21:22:38.238Z] 
[2023-04-20T21:22:38.238Z] ================= 265 passed, 3 skipped in 1398.64s (0:23:18) ==================
    
  

@bird-house bird-house deleted a comment from crim-jenkins-bot Apr 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci/operations Continuous Integration components documentation Improvements or additions to documentation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants