EESSI 2023.06 compat layer w/ OpenSSL 1.1.1 + GCC 10.4.0 (without using gcc-config) #188

boegel · 2023-06-14T06:57:26Z

This PR builds on top of the changes proposed in #184, #185, #186, joint work with @trz42 .

Complete overview of changes:

EESSI pilot version bumped to 2023.06;
mask >=dev-libs/openssl-3, so we get OpenSSL 1.1.1 in compat layer (to avoid problems with Rust, Python, Perl);
only mask >=sys-devel/gcc-11, so we get latest GCC 10.x (10.4.0) as system compiler in compat layer (since GCC 9.x seems to be no longer actively maintained in Gentoo);
stop using gcc-config to control GCC version used in compat layer, since it's not needed (there is only one GCC version thanks to masking of sys-devel/gcc), and it's used incorrectly which causes problems (see problem when installing x11-base/xorg-proto package in EESSI 2023.04 and 2023.06 #187);
update Gentoo commit to aab8473aa90e0287553b3348a5c5b17872df4b7b (8 June 2023);
Gentoo Prefix bootstrap script updated to commit c6db239 (2 June 2023);

closes #184, closes #185, closes #186

consistently bump version to 2023.06 + fix masking of OpenSSL 3.*

…42/compatibility-layer into eessi-2023.06-gentoo-2949284

… URL for our custom overlay

…e're using it incorrectly, and it's causing problems like resulting in an incomplete ld.so.conf

…een removed

eessi-bot · 2023-06-14T06:57:30Z

Instance eessi-bot-citc-aws is configured to build:

arch x86_64/generic for repo eessi-2021.12
arch x86_64/generic for repo eessi-2023.04
arch x86_64/generic for repo eessi-2023.06-compat
arch x86_64/generic for repo eessi-2023.06-software
arch x86_64/intel/haswell for repo eessi-2021.12
arch x86_64/intel/haswell for repo eessi-2023.04
arch x86_64/intel/haswell for repo eessi-2023.06-compat
arch x86_64/intel/haswell for repo eessi-2023.06-software
arch x86_64/intel/skylake_avx512 for repo eessi-2021.12
arch x86_64/intel/skylake_avx512 for repo eessi-2023.04
arch x86_64/intel/skylake_avx512 for repo eessi-2023.06-compat
arch x86_64/intel/skylake_avx512 for repo eessi-2023.06-software
arch x86_64/amd/zen2 for repo eessi-2021.12
arch x86_64/amd/zen2 for repo eessi-2023.04
arch x86_64/amd/zen2 for repo eessi-2023.06-compat
arch x86_64/amd/zen2 for repo eessi-2023.06-software
arch x86_64/amd/zen3 for repo eessi-2021.12
arch x86_64/amd/zen3 for repo eessi-2023.04
arch x86_64/amd/zen3 for repo eessi-2023.06-compat
arch x86_64/amd/zen3 for repo eessi-2023.06-software
arch aarch64/generic for repo eessi-2021.12
arch aarch64/generic for repo eessi-2023.04
arch aarch64/generic for repo eessi-2023.06-compat
arch aarch64/generic for repo eessi-2023.06-software
arch aarch64/graviton2 for repo eessi-2021.12
arch aarch64/graviton2 for repo eessi-2023.04
arch aarch64/graviton2 for repo eessi-2023.06-compat
arch aarch64/graviton2 for repo eessi-2023.06-software
arch aarch64/graviton3 for repo eessi-2021.12
arch aarch64/graviton3 for repo eessi-2023.04
arch aarch64/graviton3 for repo eessi-2023.06-compat
arch aarch64/graviton3 for repo eessi-2023.06-software

boegel · 2023-06-14T06:59:46Z

bot: build arch=x86_64/generic repo=eessi-2023.06-compat
bot: build arch=aarch64/generic repo=eessi-2023.06-compat

eessi-bot · 2023-06-14T06:59:48Z

Updates by the bot instance eessi-bot-citc-aws (click for details)

parsing the bot command build arch=x86_64/generic repo=eessi-2023.06-compat, received from sender boegel, failed
parsing the bot command build arch=aarch64/generic repo=eessi-2023.06-compat, received from sender boegel, failed

boegel · 2023-06-14T07:03:45Z

The bot should be able to build this (cfr. experiment in boegel#2), but it will complain that the build failed even though it doesn't, since the procedure to check for a correct build is still hardcoded for the software layer.

The correct tarballs will also have to created manually, because the bot/build.sh for the compat layer script tries to create the tarball outside of the build container, which doesn't work since the compat layer version that was built is not available directly via the native /cvmfs mount (which may even be missing).

This shouldn't block us from proceeding with the 2023.06 EESSI pilot version however, since we're already making good progress here (thanks to kicking out the gcc-config step, #187 is fixed, and hence the bot is able to complete the build autonomously).

trz42 · 2023-06-14T07:11:56Z

bot: build arch=x86_64/generic repo=eessi-2023.06-compat
bot: build arch=aarch64/generic repo=eessi-2023.06-compat

the following should work (= replaced with :)

bot: build arch:x86_64/generic repo:eessi-2023.06-compat
bot: build arch:aarch64/generic repo:eessi-2023.06-compat

eessi-bot · 2023-06-14T07:11:59Z

Updates by the bot instance eessi-bot-citc-aws (click for details)

received bot command build arch:x86_64/generic repo:eessi-2023.06-compat from trz42
- expanded format: build architecture:x86_64/generic repository:eessi-2023.06-compat
received bot command build arch:aarch64/generic repo:eessi-2023.06-compat from trz42
- expanded format: build architecture:aarch64/generic repository:eessi-2023.06-compat
handling command build architecture:x86_64/generic repository:eessi-2023.06-compat resulted in:
- submitted job 4939, for details & status see EESSI 2023.06 compat layer w/ OpenSSL 1.1.1 + GCC 10.4.0 (without using gcc-config) #188 (comment)
handling command build architecture:aarch64/generic repository:eessi-2023.06-compat resulted in:
- submitted job 4940, for details & status see EESSI 2023.06 compat layer w/ OpenSSL 1.1.1 + GCC 10.4.0 (without using gcc-config) #188 (comment)

eessi-bot · 2023-06-14T07:12:10Z

New job on instance eessi-bot-citc-aws for architecture x86_64-generic for repository eessi-2023.06-compat in job dir /mnt/shared/home/bot/eessi-bot-software-layer/jobs/2023.06/pr_188/4939

date	job status	comment
Jun 14 07:12:09 UTC 2023	submitted	job id `4939` awaits release by job manager
Jun 14 07:21:06 UTC 2023	released	job awaits launch by Slurm scheduler
Jun 14 07:53:17 UTC 2023	running	job `4939` is running
Jun 14 14:45:42 UTC 2023	finished	😢 FAILURE Found slurm output `slurm-4939.out` in job dir Slurm output lacks message "No missing modules!". Slurm output lacks message about created tarball. No tarball matching `eessi-\software-\.tar.gz` found in job dir.

eessi-bot · 2023-06-14T07:12:20Z

New job on instance eessi-bot-citc-aws for architecture aarch64-generic for repository eessi-2023.06-compat in job dir /mnt/shared/home/bot/eessi-bot-software-layer/jobs/2023.06/pr_188/4940

date	job status	comment
Jun 14 07:12:19 UTC 2023	submitted	job id `4940` awaits release by job manager
Jun 14 07:21:04 UTC 2023	released	job awaits launch by Slurm scheduler
Jun 14 07:53:15 UTC 2023	running	job `4940` is running
Jun 14 14:45:39 UTC 2023	finished	😢 FAILURE Found slurm output `slurm-4940.out` in job dir Slurm output lacks message "No missing modules!". Slurm output lacks message about created tarball. No tarball matching `eessi-\software-\.tar.gz` found in job dir.

trz42 · 2023-06-14T07:26:11Z

Above jobs 4939 and 4940 started but the job manager crashes when trying to update the PR comments. Possibly the PAT it uses is not valid anymore 🤷

boegel · 2023-06-14T09:38:36Z

Above jobs 4939 and 4940 started but the job manager crashes when trying to update the PR comments. Possibly the PAT it uses is not valid anymore 🤷

More details on the crash:

Traceback (most recent call last):
  File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/mnt/shared/home/bot/eessi-bot-software-layer/eessi_bot_job_manager.py", line 723, in <module>
    main()
  File "/mnt/shared/home/bot/eessi-bot-software-layer/eessi_bot_job_manager.py", line 704, in main
    job_manager.process_finished_job(known_jobs[fj])
  File "/mnt/shared/home/bot/eessi-bot-software-layer/eessi_bot_job_manager.py", line 557, in process_finished_job
    update_comment(finished_job["comment_id"], pull_request, comment_update)
  File "/mnt/shared/home/bot/eessi-bot-software-layer/tools/pr_comments.py", line 95, in update_comment
    tries=5, delay=1, backoff=2, max_delay=30)
  File "/mnt/shared/home/bot/.local/lib/python3.6/site-packages/retry/api.py", line 101, in retry_call
    return __retry_internal(partial(f, *args, **kwargs), exceptions, tries, delay, max_delay, backoff, jitter, logger)
  File "/mnt/shared/home/bot/.local/lib/python3.6/site-packages/retry/api.py", line 33, in __retry_internal
    return f()
  File "/mnt/shared/home/bot/.local/lib/python3.6/site-packages/github/IssueComment.py", line 131, in edit
    "PATCH", self.url, input=post_parameters
  File "/mnt/shared/home/bot/.local/lib/python3.6/site-packages/github/Requester.py", line 355, in requestJsonAndCheck
    verb, url, parameters, headers, input, self.__customConnection(url)
  File "/mnt/shared/home/bot/.local/lib/python3.6/site-packages/github/Requester.py", line 378, in __check
    raise self.__createException(status, responseHeaders, output)
github.GithubException.GithubException: 403 {"message": "Resource not accessible by integration", "documentation_url": "https://docs.github.com/rest/reference/issues#update-an-issue-comment"}

boegel · 2023-06-14T12:45:28Z

bot: foobar

eessi-bot · 2023-06-14T12:45:30Z

Updates by the bot instance eessi-bot-citc-aws (click for details)

received bot command foobar from boegel
- expanded format: foobar
handling command foobar failed with message
unknown command foobar; use bot: help for usage information

trz42

Looks good.

#189 can be addressed in a follow up PR.

truib and others added 17 commits June 7, 2023 10:01

bumbing version to 2023.06 and masking OpenSSL to older than v3.*

d6fb3eb

bump version to 2023.06 + mask OpenSSL versions 3.*

7c84a72

use trz42/gentoo-overlay

320111b

fix package mask for OpenSSL

be96c76

use trz42/gentoo-overlay + fix package mask for OpenSSL

970e6be

fix masking of OpenSSL 3.*

3a21636

bump EESSI version to 2023.06

7c935f9

Merge pull request #2 from boegel/eessi-2023.06-gentoo-2949284

c0bec57

consistently bump version to 2023.06 + fix masking of OpenSSL 3.*

Merge branch 'eessi-2023.06-gentoo-2949284' of https://github.com/trz…

dc18b1b

…42/compatibility-layer into eessi-2023.06-gentoo-2949284

bump Gentoo commit to June 8, 2023

0900791

use EESSI/gentoo-overlay

72bbe69

update Gentoo Prefix bootstrap script (commit c6db239, 2 June 2023)

37047e7

stick to 'EESSI/gentoo-overlay' rather than 'eessi/gentoo-overlay' in…

6e6d08a

… URL for our custom overlay

only mask GCC >= 11, to pick up fix for https://bugs.gentoo.org/865835

3032d19

set gcc 10.4.0 as default for 2023.06 compat layer

76a6767

don't run gcc-config to select GCC version to use in compat layer - w…

f7cf716

…e're using it incorrectly, and it's causing problems like resulting in an incomplete ld.so.conf

remove prefix_default_gcc, no longer used since gcc-config step has b…

7369cfe

…een removed

boegel requested a review from trz42 June 14, 2023 06:58

boegel mentioned this pull request Jun 14, 2023

add EasyBuild 4.7.2 + foss/2021a to EESSI pilot 2023.06 EESSI/software-layer#260

Merged

This was referenced Jun 14, 2023

final tarball for built compat layer not created correctly by bot/build.sh #189

Open

build compatibility layer with the bot #173

Open

trz42 approved these changes Jun 14, 2023

View reviewed changes

trz42 merged commit 2be6503 into EESSI:main Jun 15, 2023

boegel deleted the eessi-2023.06_updated_bootstrap branch June 15, 2023 09:06

This was referenced Jun 15, 2023

build version EESSI/2023.06 #185

Merged

bumping version to 2023.06 and masking OpenSSL to older than v3.* #184

Merged

boegel added the pilot-2023.06 label Jul 7, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

EESSI 2023.06 compat layer w/ OpenSSL 1.1.1 + GCC 10.4.0 (without using gcc-config) #188

EESSI 2023.06 compat layer w/ OpenSSL 1.1.1 + GCC 10.4.0 (without using gcc-config) #188

boegel commented Jun 14, 2023 •

edited

Loading

eessi-bot bot commented Jun 14, 2023

boegel commented Jun 14, 2023

eessi-bot bot commented Jun 14, 2023

boegel commented Jun 14, 2023

trz42 commented Jun 14, 2023

eessi-bot bot commented Jun 14, 2023 •

edited

Loading

eessi-bot bot commented Jun 14, 2023 •

edited

Loading

eessi-bot bot commented Jun 14, 2023 •

edited

Loading

trz42 commented Jun 14, 2023

boegel commented Jun 14, 2023

boegel commented Jun 14, 2023

eessi-bot bot commented Jun 14, 2023 •

edited

Loading

trz42 left a comment

EESSI 2023.06 compat layer w/ OpenSSL 1.1.1 + GCC 10.4.0 (without using gcc-config) #188

EESSI 2023.06 compat layer w/ OpenSSL 1.1.1 + GCC 10.4.0 (without using gcc-config) #188

Conversation

boegel commented Jun 14, 2023 • edited Loading

eessi-bot bot commented Jun 14, 2023

boegel commented Jun 14, 2023

eessi-bot bot commented Jun 14, 2023

boegel commented Jun 14, 2023

trz42 commented Jun 14, 2023

eessi-bot bot commented Jun 14, 2023 • edited Loading

eessi-bot bot commented Jun 14, 2023 • edited Loading

eessi-bot bot commented Jun 14, 2023 • edited Loading

trz42 commented Jun 14, 2023

boegel commented Jun 14, 2023

boegel commented Jun 14, 2023

eessi-bot bot commented Jun 14, 2023 • edited Loading

trz42 left a comment

Choose a reason for hiding this comment

boegel commented Jun 14, 2023 •

edited

Loading

eessi-bot bot commented Jun 14, 2023 •

edited

Loading

eessi-bot bot commented Jun 14, 2023 •

edited

Loading

eessi-bot bot commented Jun 14, 2023 •

edited

Loading

eessi-bot bot commented Jun 14, 2023 •

edited

Loading