fix flaky WPS provider responses causing failure during their registration in weaver #318

fmigneault · 2023-04-18T01:13:26Z

Overview

Fix flaky WPS provider responses (i.e.: other WPS birds) causing failure during their registration in weaver.

In some cases, the WPS birds would not respond properly when starting the stack, either because they are still initiating or due to other temporary failures such as services being restarted until healthy. This fix introduces a retry mechanism to attempt WPS registration in weaver up to WEAVER_WPS_PROVIDERS_RETRY_COUNT=5 times (1 initial attempt + 5 retries), and with WEAVER_WPS_PROVIDERS_RETRY_AFTER=5 second intervals between each retry. If the maximum number of retries for any WPS provider or the WEAVER_WPS_PROVIDERS_MAX_TIME across all registrations are reached, the operation is aborted.

…ation in weaver

crim-jenkins-bot · 2023-04-18T02:51:03Z

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/1468/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : fix-flaky-wps-provider
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-216.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1081/

NOTEBOOK TEST RESULTS

    
[2023-04-18T01:47:49.981Z] ============================= test session starts ==============================
[2023-04-18T01:47:49.981Z] platform linux -- Python 3.8.16, pytest-7.2.0, pluggy-1.0.0
[2023-04-18T01:47:49.981Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2023-04-18T01:47:49.981Z] plugins: anyio-3.6.2, dash-2.7.0, nbval-0.9.6, tornasync-0.6.0.post2
[2023-04-18T01:47:49.981Z] collected 268 items
[2023-04-18T01:47:49.981Z] 
[2023-04-18T01:47:59.734Z] notebooks-auth/test_thredds.ipynb ...........                            [  4%]
[2023-04-18T01:48:08.162Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [  6%]
[2023-04-18T01:48:15.541Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [  8%]
[2023-04-18T01:48:22.307Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 11%]
[2023-04-18T01:48:27.704Z] pavics-sdi-master/docs/source/notebooks/WPS_example.ipynb ..........     [ 15%]
[2023-04-18T01:58:48.300Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 20%]
[2023-04-18T01:58:48.300Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 20%]
[2023-04-18T01:58:51.755Z] ...............                                                          [ 26%]
[2023-04-18T01:59:03.680Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 27%]
[2023-04-18T01:59:16.368Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 30%]
[2023-04-18T01:59:38.794Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 32%]
[2023-04-18T01:59:40.704Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 32%]
[2023-04-18T01:59:58.296Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 35%]
[2023-04-18T02:00:05.421Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 37%]
[2023-04-18T02:03:39.852Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 42%]
[2023-04-18T02:04:54.952Z] .............                                                            [ 47%]
[2023-04-18T02:04:58.274Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 49%]
[2023-04-18T02:05:00.594Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 52%]
[2023-04-18T02:05:27.729Z] .................                                                        [ 58%]
[2023-04-18T02:05:34.958Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb .....           [ 60%]
[2023-04-18T02:05:36.341Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 60%]
[2023-04-18T02:05:37.691Z] .FFFFFFFF                                                                [ 64%]
[2023-04-18T02:05:49.686Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 68%]
[2023-04-18T02:06:00.152Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 70%]
[2023-04-18T02:06:41.600Z] finch-master/docs/source/notebooks/subset.ipynb ....................ss.  [ 79%]
[2023-04-18T02:06:43.510Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 79%]
[2023-04-18T02:06:54.772Z] ......                                                                   [ 81%]
[2023-04-18T02:10:31.406Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 82%]
[2023-04-18T02:11:34.652Z] .............                                                            [ 86%]
[2023-04-18T02:11:49.564Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 87%]
[2023-04-18T02:15:33.319Z] ....s.                                                                   [ 89%]
[2023-04-18T02:15:41.447Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 89%]
[2023-04-18T02:15:46.704Z] ...                                                                      [ 91%]
[2023-04-18T02:16:01.594Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 91%]
[2023-04-18T02:16:25.614Z] ......                                                                   [ 93%]
[2023-04-18T02:16:28.094Z] notebooks/hummingbird.ipynb ............                                 [ 98%]
[2023-04-18T02:20:13.782Z] notebooks/stress-tests.ipynb .....                                       [100%]
[2023-04-18T02:20:13.782Z] 
[2023-04-18T02:20:13.782Z] =================================== FAILURES ===================================

crim-jenkins-bot · 2023-04-18T02:53:10Z

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/1469/
Result : success

BIRDHOUSE_DEPLOY_BRANCH : fix-flaky-wps-provider
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-91.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1082/

NOTEBOOK TEST RESULTS

    
[2023-04-18T01:52:23.963Z] ============================= test session starts ==============================
[2023-04-18T01:52:23.963Z] platform linux -- Python 3.8.16, pytest-7.2.0, pluggy-1.0.0
[2023-04-18T01:52:23.963Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master@2
[2023-04-18T01:52:23.963Z] plugins: anyio-3.6.2, dash-2.7.0, nbval-0.9.6, tornasync-0.6.0.post2
[2023-04-18T01:52:23.963Z] collected 268 items
[2023-04-18T01:52:23.963Z] 
[2023-04-18T01:52:33.437Z] notebooks-auth/test_thredds.ipynb ...........                            [  4%]
[2023-04-18T01:52:44.913Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [  6%]
[2023-04-18T01:52:56.241Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [  8%]
[2023-04-18T01:53:06.399Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 11%]
[2023-04-18T01:53:09.519Z] pavics-sdi-master/docs/source/notebooks/WPS_example.ipynb ..........     [ 15%]
[2023-04-18T02:02:54.012Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 20%]
[2023-04-18T02:02:54.012Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 20%]
[2023-04-18T02:02:58.476Z] ...............                                                          [ 26%]
[2023-04-18T02:03:08.059Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 27%]
[2023-04-18T02:03:15.972Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 30%]
[2023-04-18T02:03:34.460Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 32%]
[2023-04-18T02:03:35.839Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 32%]
[2023-04-18T02:03:40.673Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 35%]
[2023-04-18T02:03:47.060Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 37%]
[2023-04-18T02:06:24.441Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 42%]
[2023-04-18T02:07:39.786Z] .............                                                            [ 47%]
[2023-04-18T02:07:42.225Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 49%]
[2023-04-18T02:07:44.556Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 52%]
[2023-04-18T02:08:09.870Z] .................                                                        [ 58%]
[2023-04-18T02:08:16.928Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb .....           [ 60%]
[2023-04-18T02:08:18.314Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 60%]
[2023-04-18T02:08:34.637Z] .........                                                                [ 64%]
[2023-04-18T02:08:45.141Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 68%]
[2023-04-18T02:08:55.404Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 70%]
[2023-04-18T02:09:31.093Z] finch-master/docs/source/notebooks/subset.ipynb ....................ss.  [ 79%]
[2023-04-18T02:09:33.660Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 79%]
[2023-04-18T02:09:42.069Z] ......                                                                   [ 81%]
[2023-04-18T02:13:03.641Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 82%]
[2023-04-18T02:14:25.153Z] .............                                                            [ 86%]
[2023-04-18T02:14:40.094Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 87%]
[2023-04-18T02:18:41.429Z] ....s.                                                                   [ 89%]
[2023-04-18T02:18:49.561Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 89%]
[2023-04-18T02:18:54.635Z] ...                                                                      [ 91%]
[2023-04-18T02:19:09.529Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 91%]
[2023-04-18T02:19:38.032Z] ......                                                                   [ 93%]
[2023-04-18T02:19:39.122Z] notebooks/hummingbird.ipynb ............                                 [ 98%]
[2023-04-18T02:23:13.812Z] notebooks/stress-tests.ipynb .....                                       [100%]
[2023-04-18T02:23:13.812Z] 
[2023-04-18T02:23:13.812Z] ================= 265 passed, 3 skipped in 1847.09s (0:30:47) ==================

birdhouse/components/weaver/post-docker-compose-up

…e/birdhouse-deploy into fix-flaky-wps-provider

crim-jenkins-bot · 2023-04-19T18:33:39Z

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/1495/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : fix-flaky-wps-provider
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-20.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1086/

NOTEBOOK TEST RESULTS

    
[2023-04-19T17:33:22.835Z] ============================= test session starts ==============================
[2023-04-19T17:33:22.835Z] platform linux -- Python 3.8.16, pytest-7.2.0, pluggy-1.0.0
[2023-04-19T17:33:22.835Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2023-04-19T17:33:22.835Z] plugins: anyio-3.6.2, dash-2.7.0, nbval-0.9.6, tornasync-0.6.0.post2
[2023-04-19T17:33:22.835Z] collected 268 items
[2023-04-19T17:33:22.835Z] 
[2023-04-19T17:33:32.280Z] notebooks-auth/test_thredds.ipynb ...........                            [  4%]
[2023-04-19T17:33:43.468Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [  6%]
[2023-04-19T17:33:56.135Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [  8%]
[2023-04-19T17:34:06.088Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 11%]
[2023-04-19T17:34:08.759Z] pavics-sdi-master/docs/source/notebooks/WPS_example.ipynb ..........     [ 15%]
[2023-04-19T17:45:03.996Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 20%]
[2023-04-19T17:45:06.549Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 20%]
[2023-04-19T17:45:16.593Z] ...............                                                          [ 26%]
[2023-04-19T17:45:27.235Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 27%]
[2023-04-19T17:45:38.397Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 30%]
[2023-04-19T17:45:59.208Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 32%]
[2023-04-19T17:46:01.152Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 32%]
[2023-04-19T17:46:14.381Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 35%]
[2023-04-19T17:46:19.672Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb ....F       [ 37%]
[2023-04-19T17:50:09.587Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 42%]
[2023-04-19T17:51:25.534Z] .............                                                            [ 47%]
[2023-04-19T17:51:27.983Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 49%]
[2023-04-19T17:51:30.575Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 52%]
[2023-04-19T17:52:00.545Z] .................                                                        [ 58%]
[2023-04-19T17:52:08.886Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb .....           [ 60%]
[2023-04-19T17:52:10.802Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 60%]
[2023-04-19T17:52:11.850Z] .FFFFFFFF                                                                [ 64%]
[2023-04-19T17:52:24.473Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 68%]
[2023-04-19T17:52:34.474Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 70%]
[2023-04-19T17:53:17.151Z] finch-master/docs/source/notebooks/subset.ipynb ....................ss.  [ 79%]
[2023-04-19T17:53:19.705Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 79%]
[2023-04-19T17:53:31.638Z] ......                                                                   [ 81%]
[2023-04-19T17:57:38.254Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 82%]
[2023-04-19T17:58:57.644Z] .............                                                            [ 86%]
[2023-04-19T17:59:15.788Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 87%]
[2023-04-19T18:04:07.257Z] ....s.                                                                   [ 89%]
[2023-04-19T18:04:15.417Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 89%]
[2023-04-19T18:04:21.247Z] ...                                                                      [ 91%]
[2023-04-19T18:04:36.161Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 91%]
[2023-04-19T18:05:02.353Z] ......                                                                   [ 93%]
[2023-04-19T18:05:03.448Z] notebooks/hummingbird.ipynb ............                                 [ 98%]
[2023-04-19T18:08:38.927Z] notebooks/stress-tests.ipynb .....                                       [100%]
[2023-04-19T18:08:38.927Z] 
[2023-04-19T18:08:38.927Z] =================================== FAILURES ===================================

tlvu

LGTM. Please mention in the PR description and CHANGES that the fix to flaky WPS response is to introduce a retry mechanismn. Max number of retries and retries duration are also introduced to avoid retrying forever.

fmigneault · 2023-04-19T23:45:13Z

@tlvu
see #321 replacing this PR.
Same content, but different base, so I don't integrate #284 changes yet.

crim-jenkins-bot · 2023-04-20T01:19:11Z

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/1500/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : fix-flaky-wps-provider
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-91.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1089/

NOTEBOOK TEST RESULTS

    
[2023-04-20T00:11:30.977Z] ============================= test session starts ==============================
[2023-04-20T00:11:30.977Z] platform linux -- Python 3.8.16, pytest-7.2.0, pluggy-1.0.0
[2023-04-20T00:11:30.977Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master@2
[2023-04-20T00:11:30.977Z] plugins: anyio-3.6.2, dash-2.7.0, nbval-0.9.6, tornasync-0.6.0.post2
[2023-04-20T00:11:30.977Z] collected 268 items
[2023-04-20T00:11:30.977Z] 
[2023-04-20T00:11:39.943Z] notebooks-auth/test_thredds.ipynb ...........                            [  4%]
[2023-04-20T00:11:50.176Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [  6%]
[2023-04-20T00:11:59.518Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [  8%]
[2023-04-20T00:12:07.827Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 11%]
[2023-04-20T00:12:10.696Z] pavics-sdi-master/docs/source/notebooks/WPS_example.ipynb ..........     [ 15%]
[2023-04-20T00:25:51.037Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 20%]
[2023-04-20T00:25:51.037Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 20%]
[2023-04-20T00:25:51.037Z] ...............                                                          [ 26%]
[2023-04-20T00:26:00.316Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 27%]
[2023-04-20T00:26:08.307Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 30%]
[2023-04-20T00:26:31.014Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 32%]
[2023-04-20T00:26:32.663Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 32%]
[2023-04-20T00:26:37.433Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 35%]
[2023-04-20T00:26:42.675Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb ....F       [ 37%]
[2023-04-20T00:30:52.590Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 42%]
[2023-04-20T00:32:12.427Z] .............                                                            [ 47%]
[2023-04-20T00:32:15.235Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 49%]
[2023-04-20T00:32:17.997Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 52%]
[2023-04-20T00:32:52.089Z] .................                                                        [ 58%]
[2023-04-20T00:32:59.581Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb .....           [ 60%]
[2023-04-20T00:33:00.978Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 60%]
[2023-04-20T00:33:02.332Z] .FFFFFFFF                                                                [ 64%]
[2023-04-20T00:33:15.458Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 68%]
[2023-04-20T00:33:26.336Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 70%]
[2023-04-20T00:34:15.769Z] finch-master/docs/source/notebooks/subset.ipynb ....................ss.  [ 79%]
[2023-04-20T00:34:17.683Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 79%]
[2023-04-20T00:34:33.214Z] ......                                                                   [ 81%]
[2023-04-20T00:40:09.862Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 82%]
[2023-04-20T00:41:53.586Z] .............                                                            [ 86%]
[2023-04-20T00:42:15.560Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 87%]
[2023-04-20T00:48:57.430Z] ....s.                                                                   [ 89%]
[2023-04-20T00:49:05.567Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 89%]
[2023-04-20T00:49:12.305Z] ...                                                                      [ 91%]
[2023-04-20T00:49:27.265Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 91%]
[2023-04-20T00:49:52.403Z] ......                                                                   [ 93%]
[2023-04-20T00:49:55.142Z] notebooks/hummingbird.ipynb ............                                 [ 98%]
[2023-04-20T00:53:40.774Z] notebooks/stress-tests.ipynb .....                                       [100%]
[2023-04-20T00:53:40.774Z] 
[2023-04-20T00:53:40.774Z] =================================== FAILURES ===================================

crim-jenkins-bot · 2023-04-20T01:19:12Z

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/1499/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : fix-flaky-wps-provider
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-91.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1089/

NOTEBOOK TEST RESULTS

    
[2023-04-20T00:11:30.977Z] ============================= test session starts ==============================
[2023-04-20T00:11:30.977Z] platform linux -- Python 3.8.16, pytest-7.2.0, pluggy-1.0.0
[2023-04-20T00:11:30.977Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master@2
[2023-04-20T00:11:30.977Z] plugins: anyio-3.6.2, dash-2.7.0, nbval-0.9.6, tornasync-0.6.0.post2
[2023-04-20T00:11:30.977Z] collected 268 items
[2023-04-20T00:11:30.977Z] 
[2023-04-20T00:11:39.943Z] notebooks-auth/test_thredds.ipynb ...........                            [  4%]
[2023-04-20T00:11:50.176Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [  6%]
[2023-04-20T00:11:59.518Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [  8%]
[2023-04-20T00:12:07.827Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 11%]
[2023-04-20T00:12:10.696Z] pavics-sdi-master/docs/source/notebooks/WPS_example.ipynb ..........     [ 15%]
[2023-04-20T00:25:51.037Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 20%]
[2023-04-20T00:25:51.037Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 20%]
[2023-04-20T00:25:51.037Z] ...............                                                          [ 26%]
[2023-04-20T00:26:00.316Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 27%]
[2023-04-20T00:26:08.307Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 30%]
[2023-04-20T00:26:31.014Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 32%]
[2023-04-20T00:26:32.663Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 32%]
[2023-04-20T00:26:37.433Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 35%]
[2023-04-20T00:26:42.675Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb ....F       [ 37%]
[2023-04-20T00:30:52.590Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 42%]
[2023-04-20T00:32:12.427Z] .............                                                            [ 47%]
[2023-04-20T00:32:15.235Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 49%]
[2023-04-20T00:32:17.997Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 52%]
[2023-04-20T00:32:52.089Z] .................                                                        [ 58%]
[2023-04-20T00:32:59.581Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb .....           [ 60%]
[2023-04-20T00:33:00.978Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 60%]
[2023-04-20T00:33:02.332Z] .FFFFFFFF                                                                [ 64%]
[2023-04-20T00:33:15.458Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 68%]
[2023-04-20T00:33:26.336Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 70%]
[2023-04-20T00:34:15.769Z] finch-master/docs/source/notebooks/subset.ipynb ....................ss.  [ 79%]
[2023-04-20T00:34:17.683Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 79%]
[2023-04-20T00:34:33.214Z] ......                                                                   [ 81%]
[2023-04-20T00:40:09.862Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 82%]
[2023-04-20T00:41:53.586Z] .............                                                            [ 86%]
[2023-04-20T00:42:15.560Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 87%]
[2023-04-20T00:48:57.430Z] ....s.                                                                   [ 89%]
[2023-04-20T00:49:05.567Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 89%]
[2023-04-20T00:49:12.305Z] ...                                                                      [ 91%]
[2023-04-20T00:49:27.265Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 91%]
[2023-04-20T00:49:52.403Z] ......                                                                   [ 93%]
[2023-04-20T00:49:55.142Z] notebooks/hummingbird.ipynb ............                                 [ 98%]
[2023-04-20T00:53:40.774Z] notebooks/stress-tests.ipynb .....                                       [100%]
[2023-04-20T00:53:40.774Z] 
[2023-04-20T00:53:40.774Z] =================================== FAILURES ===================================

…ure during their registration in `weaver` (#321) ## Overview Fix flaky WPS provider responses (i.e.: other WPS birds) causing failure during their registration in `weaver`. ## Changes **Non-breaking changes** - Fix flaky WPS provider responses (i.e.: other WPS birds) causing failure during their registration in `weaver`. In some cases, the WPS birds would not respond properly when starting the stack, either because they are still initiating or due to other temporary failures such as services being restarted until healthy. This fix introduces a retry mechanism to attempt WPS registration in `weaver` up to `WEAVER_WPS_PROVIDERS_RETRY_COUNT=5` times (1 initial attempt + 5 retries), and with `WEAVER_WPS_PROVIDERS_RETRY_AFTER=5` second intervals between each retry. If the maximum number of retries for any WPS provider or the `WEAVER_WPS_PROVIDERS_MAX_TIME` across all registrations are reached, the operation is aborted. **Breaking changes** - n/a ## Related Issue / Discussion - Close #318 (using `fix-canarieapi` as base branch) instead of `master` (this PR)

fmigneault added 4 commits April 17, 2023 21:11

fix flaky WPS provider responses causing failure during their registr…

beb44c0

…ation in weaver

patch for loop break

2ed1ba7

add missing prefix

073fffc

fix broken pipe error

09697ea

bird-house deleted a comment from crim-jenkins-bot Apr 18, 2023

fmigneault mentioned this pull request Apr 18, 2023

fix CanarieAPI config to support Python 3.x #284

Merged

mishaschwartz reviewed Apr 18, 2023

View reviewed changes

birdhouse/components/weaver/post-docker-compose-up Outdated Show resolved Hide resolved

birdhouse/components/weaver/post-docker-compose-up Show resolved Hide resolved

fmigneault added 3 commits April 18, 2023 15:31

Merge branch 'fix-canarieapi' into fix-flaky-wps-provider

b91e5d8

update random number broken pipe handling strategy

9709bd2

Merge branch 'fix-flaky-wps-provider' of https://github.com/bird-hous…

c9a032e

…e/birdhouse-deploy into fix-flaky-wps-provider

bird-house deleted a comment from crim-jenkins-bot Apr 18, 2023

fmigneault added 3 commits April 18, 2023 23:46

Merge branch 'fix-canarieapi' into fix-flaky-wps-provider

ace64af

remove extra comma causing tuple as release time in canarieapi configs

fef989e

Merge branch 'fix-canarieapi' into fix-flaky-wps-provider

7065629

bird-house deleted a comment from crim-jenkins-bot Apr 19, 2023

fmigneault force-pushed the fix-canarieapi branch from fef989e to eea7230 Compare April 19, 2023 17:11

bird-house deleted a comment from crim-jenkins-bot Apr 19, 2023

Merge branch 'fix-canarieapi' into fix-flaky-wps-provider

7799843

fmigneault requested a review from tlvu April 19, 2023 19:01

fmigneault self-assigned this Apr 19, 2023

tlvu approved these changes Apr 19, 2023

View reviewed changes

fmigneault changed the base branch from fix-canarieapi to master April 19, 2023 23:41

fmigneault changed the base branch from master to fix-canarieapi April 19, 2023 23:42

fmigneault mentioned this pull request Apr 19, 2023

Fix flaky WPS provider responses (i.e.: other WPS birds) causing failure during their registration in weaver #321

Merged

fmigneault closed this in #321 Apr 20, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix flaky WPS provider responses causing failure during their registration in weaver #318

fix flaky WPS provider responses causing failure during their registration in weaver #318

fmigneault commented Apr 18, 2023 •

edited

Loading

crim-jenkins-bot commented Apr 18, 2023

crim-jenkins-bot commented Apr 18, 2023

crim-jenkins-bot commented Apr 19, 2023

tlvu left a comment

fmigneault commented Apr 19, 2023

crim-jenkins-bot commented Apr 20, 2023

crim-jenkins-bot commented Apr 20, 2023

fix flaky WPS provider responses causing failure during their registration in weaver #318

fix flaky WPS provider responses causing failure during their registration in weaver #318

Conversation

fmigneault commented Apr 18, 2023 • edited Loading

Overview

crim-jenkins-bot commented Apr 18, 2023

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

crim-jenkins-bot commented Apr 18, 2023

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

crim-jenkins-bot commented Apr 19, 2023

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

tlvu left a comment

Choose a reason for hiding this comment

fmigneault commented Apr 19, 2023

crim-jenkins-bot commented Apr 20, 2023

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

crim-jenkins-bot commented Apr 20, 2023

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

fmigneault commented Apr 18, 2023 •

edited

Loading