set_libraries!() fails on fresh install

[johnomotani]

When I try to run

using HDF5
HDF5.API.set_libraries!(joinpath(hdf5_dir, "libhdf5.so"),
                        joinpath(hdf5_dir, "libhdf5_hl.so"))

after starting with an empty .julia directory and with a new project, I often (always?) get an error like

ERROR: LoadError: Cannot set preferences of an unknown package that is not loaded!
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:35
 [2] set_preferences!(::Base.UUID, ::Pair{String, String}, ::Vararg{Pair{String, String}}; export_prefs::Bool, active_project_only::Bool, kwargs::@Kwargs{})
   @ Preferences ~/.julia/packages/Preferences/TTEAN/src/Preferences.jl:263
 [3] set_preferences!
   @ ~/.julia/packages/Preferences/TTEAN/src/Preferences.jl:205 [inlined]
 [4] set_libraries!(libhdf5::String, libhdf5_hl::String; force::Bool)
   @ HDF5.API ~/.julia/packages/HDF5/Ws1wH/src/api/api.jl:55
 [5] set_libraries!(libhdf5::String, libhdf5_hl::String)
   @ HDF5.API ~/.julia/packages/HDF5/Ws1wH/src/api/api.jl:37
 [6] top-level scope
   @ .../setup_script.jl:107

[I've edited out the full file paths as irrelevant]

If I explicitly install (i.e. Pkg.add()) HDF5_jll as well as HDF5 then this error goes away, but I don't think I should have to do that, because HDF5_jll is a dependency of HDF5. Maybe HDF5_jll needs to be imported somewhere so that set_libraries!() can see it??

Edit to add: This was using julia-1.10.0, HDF5.jl-0.17.1, and HDF5_jll.jl-1.12.2+2

[mkitti]

@JoshuaLampert, can you take a look?

My current thought is that we probably should add HDF5_jll to the project if we are going to set a preference on it. The main question is can we do that for the user or not?

Before Julia 1.11 (master branch), adding a dependency on Pkg was basically free, but from Julia 1.11, Pkg can be quite the dependency to load.

My main resolution here would be to add a friendlier message about adding HDF5_jll.jl to the project. Speculatively, we might be able to do the following for the user.

@eval Main begin
    using Pkg
    Pkg.add("HDF5_jll") # include UUID
end

[johnomotani]

If this (having to Pkg.add("HDF5_jll")) is expected behavour, just having an error message saying this instead of the rather cryptic one in my original post would be good, and might be enough?

For me, it's not a problem to add HDF5_jll as long as I know that I'm supposed to have to do that, and it's not just a workaround for a bug.

[JoshuaLampert]

IMO, it should also be possible without explicitly adding HDF5_jll. I look into it. But as a workaround you can add HDF5_jll to your project.

[JoshuaLampert]

I cannot reproduce this. With an empty ~/.julia directory in an empty project on julia 1.10.0:

julia> using Pkg

julia> Pkg.activate(".")
  Activating new project at `~/test_hdf5_set_libraries`

julia> Pkg.status()
Status `~/test_hdf5_set_libraries/Project.toml` (empty project)

julia> Pkg.add("HDF5")
  Installing known registries into `~/.julia`
  [...]

julia> using HDF5

julia> hdf5_dir = "/usr/lib/x86_64-linux-gnu/hdf5/openmpi/"
"/usr/lib/x86_64-linux-gnu/hdf5/openmpi/"

julia> HDF5.API.set_libraries!(joinpath(hdf5_dir, "libhdf5.so"),
                               joinpath(hdf5_dir, "libhdf5_hl.so"))
[ Info: Please restart Julia and reload HDF5.jl for the library changes to take effect

EDIT: For me, HDF5_jll is added to the [extras] in the Project.toml when HDF5.API.set_libraries! is called.

EDIT: As long as there are no preferences set, we load HDF5_jll and thus Preferences.jl finds it here. Once we have set the preferences, we have it in the Project.toml [extras] and it is found here.

[JoshuaLampert]

Did you have any LocalPreferences.toml in your active project before, @johnomotani? If yes, it should also work again after deleting the LocalPreferences.toml.

[johnomotani]

@JoshuaLampert thanks, yes it does work after deleting LocalPreferences.toml.

Sorry, I didn't work hard enough at making a reproducible example to start with! If I do what you did, but then delete everything in Project.toml while leaving LocalPreferences.toml there, then I get the error in the original post. Deleting LocalPreferences.toml fixes the problem. Doing Pkg.add("HDF5_jll") also fixes the problem.

[JoshuaLampert]

Yes, that makes sense. The problem only occurs when there are preferences in the LocalPreferences.toml and there is no HDF5_jll in the Project.toml. This is kind of an edge case since it only occurs when copying a LocalPreferences.toml file into a project or when manually deleting HDF5_jll from the Project.toml after setting the preferences, but not when setting the preferences normally. Should we catch this case anyway? If yes, I see two possibilities to deal with the situation:

1. Give a more helpful error message that suggests to either delete the LocalPreferences.toml or Pkg.adding HDF5_jll.
2. Make it work. For this, we have again some possibilities. Either Pkg.add("HDF5_jll") as suggested by @mkitti above, manually write HDF5_jll to the [extras] section of the Project.toml similar to what Preferences.jl does, or using HDF5_jll (which also prevents Preferences.jl to raise the error). I'm not quite sure if the last two possibilities are feasible and don't have any negative side effects.

What are your thoughts on this, @mkitti?