Create SSH access replacements for calls to docker.exec_run() #362

ckunki · 2023-07-14T11:20:26Z

Closes #304

All Submissions:

Is the title of the Pull Request correct?
Is the title of the corresponding issue correct?
Have you updated the changelog?
Have you checked to ensure there aren't other open Pull Requests for the same update/change?
Are you mentioning the issue which this PullRequest fixes ("Fixes...")
Before you merge don't forget to run all tests for all Exasol version, by adding [run all tests] to the commit message
Are the CLI usage examples up to date?

incl. unit tests

ported test_upload.py to pytest fixed test container names for parameterized tests

parameterized test_db_available()

…or_calls_to_docker.exec_run # Conflicts: # doc/changes/changes_2.0.0.md

exasol_integration_test_docker_environment/lib/base/db_os_executor.py

Co-authored-by: Torsten Kilias <[email protected]>

exasol_integration_test_docker_environment/lib/base/db_os_executor.py

Moved instanciate DockerClient to DockerExecFactory.executor() Replaced interface DockerClientFactory by implementation based on ContextDockerClientFactory

exasol_integration_test_docker_environment/lib/base/db_os_executor.py

...r_environment/lib/test_environment/database_setup/docker_db_log_based_bucket_sync_checker.py

..._test_docker_environment/lib/test_environment/database_setup/find_exaplus_in_db_container.py

...r_environment/lib/test_environment/database_setup/docker_db_log_based_bucket_sync_checker.py

test/unit/test_db_os_executor.py

Co-authored-by: Torsten Kilias <[email protected]>

…_docker.exec_run' of https://github.com/exasol/integration-test-docker-environment into feature/304-Create_SSH_access_replacements_for_calls_to_docker.exec_run

…ation-test-docker-environment into feature/304_fix_tk # Conflicts: # exasol_integration_test_docker_environment/lib/base/db_os_executor.py

tkilias

I didn't see, if we have a integration test for the executor which only run these

exasol_integration_test_docker_environment/lib/base/db_os_executor.py

...ion_test_docker_environment/lib/test_environment/database_waiters/db_container_log_thread.py

...on_test_docker_environment/lib/test_environment/database_waiters/is_database_ready_thread.py

...ration_test_docker_environment/lib/test_environment/spawn_test_environment_with_docker_db.py

Co-authored-by: Torsten Kilias <[email protected]>

…_docker.exec_run' of https://github.com/exasol/integration-test-docker-environment into feature/304-Create_SSH_access_replacements_for_calls_to_docker.exec_run # Conflicts: # exasol_integration_test_docker_environment/lib/base/db_os_executor.py

ckunki · 2023-07-21T10:04:05Z

I didn't see, if we have a integration test for the executor which only run these

I (again) admire your razor 🔪 eyes. 👀

We have

integration tests test/integration/test_ssh_access.py which verify actual SSH access
and unit tests test/unit/test_db_os_executor.py which verify the basic working of the executors and factories.

Up to now we do not have a dedicated integration test for executors and factories ...

... but we have parameterized tests in test/integration/test_bucketfs_upload.py which will cover full integration tests involving executors and factories as well, as soon as the implementation will be updated to actually use them. This is planned in ticket #305.

So if I can convince you to follow my explanation we could go this way.

Otherwise it would not bee too much effort to add dedicated integration tests for executors and factories. The only downside would be some additional runtime required in the CI build.

tkilias · 2023-07-21T12:39:16Z

Otherwise it would not bee too much effort to add dedicated integration tests for executors and factories. The only downside would be some additional runtime required in the CI build.

Hm, I am thinking about, if we can make these integration tests cheaper. In the sense, do we really need a database to test this functionality. I think, any ssh capable docker container should work.

ckunki · 2023-07-21T14:08:00Z

Hm, I am thinking about, if we can make these integration tests cheaper. In the sense, do we really need a database to test this functionality. I think, any ssh capable docker container should work.

OK: As aligned:

I will add an integration test using docker container linuxserver/openssh-server:9.3_p2-r0-ls123
compare script-languages/paramiko_connection_test.py

client.images.pull("panubo/sshd", tag="1.1.0")
container = client.containers.run(
    name="ssshd", 
    image="linuxserver/openssh-server:9.3_p2-r0-ls123",
    environment=[...])

Appropriate environment variables (see documentation on docker hub):

PUBLIC_KEY
PUBLIC_KEY_FILE

exasol_integration_test_docker_environment/lib/base/db_os_executor.py

ckunki added 8 commits July 10, 2023 14:27

added DB OS factories and executors

bba270e

incl. unit tests

renamed variable test_container to container

8b4a6d7

Added support for accessing database OS via SSH

da184d5

ported test_upload.py to pytest fixed test container names for parameterized tests

ported test_cli_test_environment.py to pytest

aff53d6

parameterized test_db_available()

Updated changes file

54cf33f

Merge branch 'main' into feature/304-Create_SSH_access_replacements_f…

0d164e1

…or_calls_to_docker.exec_run # Conflicts: # doc/changes/changes_2.0.0.md

reverted version to 2.0.0

d5a78f8

fixed test

a688bb9

tkilias requested changes Jul 17, 2023

View reviewed changes

ckunki and others added 2 commits July 17, 2023 13:23

Apply suggestions from code review

4d43ab2

Co-authored-by: Torsten Kilias <[email protected]>

Fixed findings

c955f82

ckunki requested a review from tkilias July 17, 2023 13:11

tkilias reviewed Jul 17, 2023

View reviewed changes

exasol_integration_test_docker_environment/lib/base/db_os_executor.py Outdated Show resolved Hide resolved

ckunki added 2 commits July 17, 2023 17:15

Fixed additional review findings:

d2a1b76

Moved instanciate DockerClient to DockerExecFactory.executor() Replaced interface DockerClientFactory by implementation based on ContextDockerClientFactory

Added tests for factory and client.close()

db10083

ckunki requested a review from tkilias July 17, 2023 15:46

fixed review findings

93ef75b

tkilias reviewed Jul 18, 2023

View reviewed changes

exasol_integration_test_docker_environment/lib/base/db_os_executor.py Outdated Show resolved Hide resolved

tkilias reviewed Jul 18, 2023

View reviewed changes

exasol_integration_test_docker_environment/lib/base/db_os_executor.py Outdated Show resolved Hide resolved

fixed review findings

bcaf4a7

tkilias requested changes Jul 18, 2023

View reviewed changes

ckunki and others added 9 commits July 18, 2023 13:04

fixed review finding

a2a0947

Apply suggestions from code review

a66a914

Co-authored-by: Torsten Kilias <[email protected]>

Merge branch 'feature/304-Create_SSH_access_replacements_for_calls_to…

93afda6

…_docker.exec_run' of https://github.com/exasol/integration-test-docker-environment into feature/304-Create_SSH_access_replacements_for_calls_to_docker.exec_run

Fixed some additional review findings

5841656

Replaced Mock by create_autospec

82af5fe

removed obsolete @runtime_checkable

7b1e7ec

used mock_cast instead of assert_called()

7ec7258

Added missing import

74b55d0

comma

4629d40

ckunki requested a review from tkilias July 19, 2023 08:56

ckunki and others added 4 commits July 19, 2023 16:51

fixed context for client factory

3af5746

Surround run method in threads with try except and logging

676f789

Merge branch 'feature/304_fix_tk' of https://github.com/exasol/integr…

320eada

…ation-test-docker-environment into feature/304_fix_tk # Conflicts: # exasol_integration_test_docker_environment/lib/base/db_os_executor.py

removed debugging aids

0748dbb

ckunki force-pushed the feature/304-Create_SSH_access_replacements_for_calls_to_docker.exec_run branch from 3731749 to 0748dbb Compare July 19, 2023 15:15

fixed test

5e4e49b

tkilias requested changes Jul 20, 2023

View reviewed changes

ckunki and others added 6 commits July 20, 2023 15:54

fixed review finding

3bd6b44

Apply suggestions from code review

7d94bd6

Co-authored-by: Torsten Kilias <[email protected]>

fixed review finding #2

0f01249

removed print commands

8e9ce3a

fixed review finding

c4dfb32

ckunki added 2 commits July 24, 2023 09:15

Added integration test for DbOsExecFactory

a539829

Created custom fixture to feed empty string to stdin

a4c4027

tkilias previously approved these changes Jul 24, 2023

View reviewed changes

exasol_integration_test_docker_environment/lib/base/db_os_executor.py Show resolved Hide resolved

ckunki added 2 commits July 24, 2023 13:30

[run all tests]

9c7d5e2

Needed to remove test_cli_test_environment.py from minimal tests

1a6a967

ckunki dismissed tkilias’s stale review via 1a6a967 July 24, 2023 13:32

ckunki added 4 commits July 24, 2023 14:32

[run all tests]

d1a4a0f

updated minimal_tests

8c68c13

updated minimal_tests

ad0caf1

[run all tests]

1dc0efe

tkilias approved these changes Jul 24, 2023

View reviewed changes

ckunki merged commit 7c0e930 into main Jul 24, 2023

ckunki deleted the feature/304-Create_SSH_access_replacements_for_calls_to_docker.exec_run branch July 24, 2023 15:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create SSH access replacements for calls to docker.exec_run() #362

Create SSH access replacements for calls to docker.exec_run() #362

ckunki commented Jul 14, 2023 •

edited

Loading

tkilias left a comment

ckunki commented Jul 21, 2023 •

edited

Loading

tkilias commented Jul 21, 2023

ckunki commented Jul 21, 2023

Create SSH access replacements for calls to docker.exec_run() #362

Create SSH access replacements for calls to docker.exec_run() #362

Conversation

ckunki commented Jul 14, 2023 • edited Loading

All Submissions:

tkilias left a comment

Choose a reason for hiding this comment

ckunki commented Jul 21, 2023 • edited Loading

tkilias commented Jul 21, 2023

ckunki commented Jul 21, 2023

ckunki commented Jul 14, 2023 •

edited

Loading

ckunki commented Jul 21, 2023 •

edited

Loading