Add integration test framework #492

mishaschwartz · 2025-01-07T14:22:25Z

Overview

This update adds a framework for testing the deployed stack using pytest. This will allow developers to check
that their changes are consistent with the existing stack and to add additionally tests when new functionality
is introduced.

Changes to implement this include:

existing unit tests are moved to the tests/unit/ directory
new integration tests are written in the tests/integration/ directory. More tests will be added in the
future!
conftest.py scripts updated to bring the stack up/down in a consistent way for the integration tests.
unit tests updated to accomodate new testing infrastructure as needed.
unit tests updated to test logging outputs better
birdhouse interface script updated to support testing infrastructure (this should not change anything for
other end-users).
additional documentation added to birdhouse interface to improve user experience.
docker healthchecks added to more components so that the readiness of the stack can be determined with or
without the use of the canarie-api component.

Next steps:

add more integration tests as needed
add a framework for testing migrating the stack from one version to another

Changes

Non-breaking changes

Adds integration tests and healthchecks

Breaking changes

None

Related Issue / Discussion

Introduces a mechanism for testing the stack locally so that we don't need to rely on integration tests run by notebooks through Jenkins. Since the Jenkins CI tests have been broken lately.
This also gives developers a mechanism to tests their work locally and establishes a simple way to write integration tests (without the need to wrap them in jupyter notebooks).
This PR only introduces the framework for creating these tests. Once this is approved, the actual work of writing the tests can start.
Further discussion/planning is needed if we want to eventually run these tests in the CI pipeline

Additional Information

CI Operations

birdhouse_daccs_configs_branch: master
birdhouse_skip_ci: false

tlvu

Very interesting PR. Just my just pass. I'll come back.

birdhouse/birdhouse-compose.sh

birdhouse/components/geoserver/docker-compose-extra.yml

tlvu · 2025-01-09T19:55:31Z

tests/env.local.test

+    ./optional-components/local-dev-test
+    ./optional-components/prometheus-log-parser
+    ./optional-components/secure-data-proxy
+    ./optional-components/wps-healthchecks


Do we still need this ./optional-components/wps-healthchecks? Should it be deleted given you added health check to all containers by default (good call).

I left the WPS healthchecks in their own optional component. I'm happy to move them into the main docker compose files so that they're enabled by default but I left them as they were initially.

Not for this PR if you prefer separate, but I really like your idea to have all healthcheck on by default instead of optionally. I do not see any good reasons for someone to disable it so why make it optional.

Yeah I think I agree. I'll see if @fmigneault has an opinion but otherwise I can merge the WPS healthchecks in as part of this PR

I'm fine with making them defined by default.

tlvu

one more minor comment

tlvu · 2025-01-15T00:04:45Z

birdhouse/scripts/logging.include.sh

@@ -1,6 +1,10 @@
 #!/bin/sh

-BIRDHOUSE_COLOR=${BIRDHOUSE_COLOR:-1}
+# support NO_COLOR flag (https://no-color.org/)
+if [ -z "${BIRDHOUSE_COLOR}" ] && [ "${NO_COLOR}" != '1' ]; then


Suggested change

if [ -z "${BIRDHOUSE_COLOR}" ] && [ "${NO_COLOR}" != '1' ]; then

if [ -z "${BIRDHOUSE_COLOR}" ] && [ -z "${NO_COLOR}" ]; then

https://no-color.og/ says "NO_COLOR environment variable that, when present and not an empty string (regardless of its value), prevents the addition of ANSI color." so I think we should not just check for NO_COLOR == 1.

fmigneault

Nice PR!

birdhouse/components/geoserver/docker-compose-extra.yml

fmigneault · 2025-01-15T02:11:52Z

bin/birdhouse

+  if [ "${option}" = "no-suffix" ]; then
+    configs_cmd_suffix=
+  fi


If $(print_config_command) was used, wouldn't if [ -z "$1" ]; then work the same?

Just curious. no-suffix is fine, but seems like a roundabout way with unnecessary hardcoded values.

fmigneault · 2025-01-15T02:17:47Z

birdhouse/components/proxy/docker-compose-extra.yml

@@ -22,3 +22,10 @@ services:
      REQUESTS_CA_BUNDLE: /etc/ssl/certs/ca-certificates.crt
    restart: always
    logging: *default-logging
+    healthcheck:
+      test: ["CMD", "curl", "--fail", "${BIRDHOUSE_FQDN_PUBLIC}"]


Ideally we should use an internal endpoint/port to avoid loop-back requests.

Also, consider using another endpoint such as the /version, since the root path gets redirected to whichever "frontend" entrypoint is defined, which might be inconsistent or pinging another redirected service from the stack.

fmigneault · 2025-01-15T02:27:24Z

tests/env.local.test

Can this be renamed to test.env so highlights are properly done on GitHub?
This will help read and review changes.

We can add a top-level comment to be explicit that this corresponds to env.local corresponding to https://github.com/bird-house/birdhouse-deploy/blob/master/birdhouse/env.local.example.

fmigneault · 2025-01-15T02:28:11Z

tests/env.local.test

+    ./optional-components/local-dev-test
+    ./optional-components/prometheus-log-parser
+    ./optional-components/secure-data-proxy
+    ./optional-components/wps-healthchecks


I'm fine with making them defined by default.

tlvu

Other than the minor NO_COLOR left, great PR !

tlvu · 2025-01-15T03:56:53Z

tests/integration/conftest.py

+
+
+@pytest.fixture(scope="module", autouse=True)
+def start_stack(request, cli_path, local_env_file, tmp_data_persist_root, stack_info, pytestconfig):


I haven't seen usage of this new fixture. In fact, same for most of the new fixtures here.

I am guessing you are simply setting up for the future?

mishaschwartz added 3 commits January 6, 2025 12:07

initial integration tests

da9d034

Merge branch 'master' into ci-tests

f9de4bf

add changes and formatting updates

8b7e875

mishaschwartz requested review from tlvu and fmigneault as code owners January 7, 2025 14:22

mishaschwartz added 2 commits January 7, 2025 09:26

fix TERM required in logging tests

f6a61d4

fix tests that only fail when TERM is unset

e9282cc

tlvu reviewed Jan 9, 2025

View reviewed changes

tlvu self-requested a review January 9, 2025 20:02

add restart to template build trigger

78b6e14

tlvu reviewed Jan 15, 2025

View reviewed changes

tlvu self-requested a review January 15, 2025 00:26

fmigneault reviewed Jan 15, 2025

View reviewed changes

tlvu approved these changes Jan 15, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add integration test framework #492

Add integration test framework #492

mishaschwartz commented Jan 7, 2025

tlvu left a comment

tlvu Jan 9, 2025

mishaschwartz Jan 10, 2025

tlvu Jan 10, 2025

mishaschwartz Jan 10, 2025

fmigneault Jan 15, 2025

tlvu left a comment

tlvu Jan 15, 2025

tlvu Jan 15, 2025

fmigneault left a comment

fmigneault Jan 15, 2025

fmigneault Jan 15, 2025

fmigneault Jan 15, 2025

fmigneault Jan 15, 2025

tlvu left a comment

tlvu Jan 15, 2025

	if [ -z "${BIRDHOUSE_COLOR}" ] && [ "${NO_COLOR}" != '1' ]; then
	if [ -z "${BIRDHOUSE_COLOR}" ] && [ -z "${NO_COLOR}" ]; then



		@pytest.fixture(scope="module", autouse=True)
		def start_stack(request, cli_path, local_env_file, tmp_data_persist_root, stack_info, pytestconfig):

Add integration test framework #492

Are you sure you want to change the base?

Add integration test framework #492

Conversation

mishaschwartz commented Jan 7, 2025

Overview

Changes

Related Issue / Discussion

Additional Information

CI Operations

tlvu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tlvu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fmigneault left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tlvu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment