type does not exist in workspace; reconstructing ... when loading interpolant from scratchspace new on 1.11 vs. 1.10

[70Gage70]

I'm not sure if this is happening because of Scratch.jl, Interpolations.jl or JLD2.jl, but the error is raised by JLD2.jl so I'm posting my MWE here. See the following little package

src

module ItpScratchJLD
export BAR, create_itp, load_itp

using Scratch, JLD2, Interpolations

struct MyStruct{I}
    itp::I
end

PATH_SCRATCH = Ref{String}()
BAR = Ref{MyStruct}()

function create_itp(; path = PATH_SCRATCH.x)
    outfile = joinpath(path, "itp.jld2")
    rm(outfile, force = true)
    itp = cubic_spline_interpolation(1:10, 1:10) |> MyStruct
    jldsave(outfile, itp = itp)
end

function load_itp(; path = PATH_SCRATCH.x)
    BAR.x = load(joinpath(path, "itp.jld2"), "itp")
end

function __init__()
    PATH_SCRATCH.x = @get_scratch!("TEST")

    try
        load_itp()
    catch
        nothing
    end
end

end # module

Project.toml

name = "ItpScratchJLD"
uuid = "f4526136-8e65-11ef-0aeb-f95001e491ea"
authors = ["Test"]
version = "0.0.1"

[deps]
Interpolations = "a98d9a8b-a2ab-59e6-89dd-64a1c18fca59"
JLD2 = "033835bb-8acc-5ee8-8aae-3f567f8a3819"
Scratch = "6c6a2e73-6563-6170-7368-637461726353"

This (roughly) reproduces the workflow of a much larger package, where interpolants are created and saved to a scratchspace using JLD2.jl so that they can be loaded later without having to recompute everything. If the interpolants exist, they are loaded automatically by __init__().

On 1.10.5, if you using ItpScratchJLD, then run create_itp(), then close everything, then using ItpScratchJLD again, you find that BAR is set correctly.

Now doing the same thing on 1.11.1, the warning is raised and BAR is not loaded, although it can be loaded afterwards with no issue.

WARNING MESSAGE TEXT

┌ Warning: type ItpScratchJLD.MyStruct{Interpolations.Extrapolation{Float64, 1, Interpolations.ScaledInterpolation{Float64, 1, Interpolations.BSplineInterpolation{Float64, 1, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, Interpolations.BSpline{Interpolations.Cubic{Interpolations.Line{Interpolations.OnGrid}}}, Tuple{Base.OneTo{Int64}}}, Interpolations.BSpline{Interpolations.Cubic{Interpolations.Line{Interpolations.OnGrid}}}, Tuple{UnitRange{Int64}}}, Interpolations.BSpline{Interpolations.Cubic{Interpolations.Line{Interpolations.OnGrid}}}, Interpolations.Throw{Nothing}}} does not exist in workspace; reconstructing
└ @ JLD2 ~/.julia/packages/JLD2/IyyZz/src/data/reconstructing_datatypes.jl:577

Very tricky.

[JonasIsensee]

Hi @70Gage70,

thanks for the report. I ran your examples and can reproduce.
Since everything works on v1.10, and on v1.11 as well (as long as it is not executed during package loading)
I don't think this is really a JLD2 issue but more on the julia side.
I tested that without __init__ you can also load the data on first try even on v1.11.

Here's a further reduced problem: (remove Interpolations, make MyStruct not be parametric and just store something like an Int)

julia> using ItpScratchJLD
┌ Warning: type ItpScratchJLD.MyStruct does not exist in workspace; reconstructing

So it seems that the module ItpScratchJLD is no longer added to Main before running __init__

[70Gage70]

@JonasIsensee thanks for your comments. I did a quick search on open Julia issues related to __init__, it looks like my issue is possibly a subset of this one. I confirmed that pkgdir prints nothing on 1.11.1 but the correct directory on 1.10.5.

[thchr]

I have a similar problem. I load some data inside __init__ which contains structs defined in the package itself. This previously worked fine (e.g., on 1.10) but on v1.11, JLD2 cannot recognize the types - even though they are defined in the package the __init__ is being called from.

[thchr]

Can confirm that this works on Julia v1.11.0-rc1, but not on later versions - so seems very likely to be due to the above-linked Julia issue, cf. this comment JuliaLang/julia#56077 (comment).

[70Gage70]

This seems to be fixed on JLD2 v0.5.10 + Julia 1.11.2?

[JonasIsensee]

Great that it works for you.
As described above, there were no changes to JLD2 to make this possible and I don't see what would have changed in Julia, either..

[70Gage70]

Yes, I'm not really sure what's going on...

@thchr Have the latest versions of JLD2 + Julia fixed the issue in your use case?

[thchr]

No, I checked on v1.11.2 earlier today and I'm getting the same issue as before unfortunately :(.

[70Gage70]

Okay, maybe this issue should stay open for now then.