Changed requirements in HDF5_jll's libhdf5.so for libcurl.so?

[sloede]

Up until recently (say 1-2 weeks ago), we always had to set

LD_PRELOAD=/lib/x86_64-linux-gnu/libcurl.so.4

in our GitHub Action workflow when using the locally installed /usr/lib/x86_64-linux-gnu/hdf5/openmpi/libhdf5.so in the GitHub runner. Otherwise, we would get an error such as

/opt/hostedtoolcache/julia/1.9.3/x64/bin/../lib/julia/libcurl.so: version `CURL_OPENSSL_4' not found (required by /usr/lib/x86_64-linux-gnu/hdf5/openmpi/libhdf5.so)

(see, e.g., here).

This LD_PRELOAD funny business was OK-ish, since it was only required for CI jobs on GitHub runners and it also worked with the JLL-provided libhdf5.so library.

However, since a few days ago, preloading the system libcurl.so does not work anymore with the JLL-provided libhdf5.so. If we do that, we get the following error:

/lib/x86_64-linux-gnu/libcurl.so.4: version `CURL_4' not found (required by /home/runner/.julia/artifacts/2829a1f6a9ca59e5b9b53f52fa6519da9c9fd7d3/lib/libhdf5.so)

(see, e.g., here).

Does anyone know where this change is coming from and/or whether there is a way around this? The trouble this causes us is that in our CI scripts, we now have to install HDF5 without LD_PRELOAD being set, then configure our project to use the local libhdf5.so, then set LD_PRELOAD, then disable it again when we run some Julia tests (which use the JLL libraries), and then set it again once we run our parallel tests with the local MPI+HDF5 stack.

cc @bgeihe

[lj-cug]

I have also encountered this problem. I cannot solve it now.

[mkitti]

Can you narrow down the version numbers are involved?

Does this occur with any version of HDF5.jl? What if you try HDF5.jl 0.16.15?

What versions of HDF5_jll does this occur for?

Libcurl is needed by HDF5 for the the read-only S3 (ROS3) driver.

[lj-cug]

After using lower version JDF5.jl, this problem disappeared. Thank you!

[mkitti]

Can you tell us what versions resolved the issue?

[lj-cug]

At least, I use "add [email protected]", there is no problem. If I use "add HDF5", the newest version was used, the above issue happened. The reason is that the Libcurl library version is low, which was not suitable for julia. I don't understand the "read-only S3 (ROS3) driver" for HDF5. I always use HDF5 in my C or FORTRAN codes. This is the first time to use HDF5 in julia.

[mkitti]

Basically, the ROS3 driver let's the HDF5 library talk to a HDF5 file stored in Amazon S3. This requires libcurl.

I'm guessing that the operating system libcurl uses OpenSSL and thus uses symbol CURL_OPENSSL_4 rather than CURL_4

[lj-cug]

That's right! The error printout shows me the libcurl uses OpenSSL. Should I install lincurl using other SSL configuration? Maybe this can solve the problem.

[mkitti]

Julia's libcurl recipe is here:
https://github.com/JuliaPackaging/Yggdrasil/blob/master/L/LibCURL/common.jl

It usually uses mbedTLS rather than OpenSSL.