v0.14 release tracker

[musm]

I propose tagging a major release by the end of the week sans objections?

Checklist before release:

• Decide what to un-export.
• Move low-level method specializations to the middle-level API #723
• Grab bag of cleaning up #724
• 3-arg show with hiearchical tree printing #725 (pretty printing) (optional?)
• Rename public functions and stop exporting several public API methods #699 (rename functions)

Save for next major release (note this can come soon after if we are ready)?

• RFC: Initialize runtime globals during __init__ #703

[musm]

After tagging, we can then fix JLD and MAT?

[kleinhenz]

Do JLD and MAT have compat bounds so that our release won't break them?

[musm]

Yes they do

[jmert]

As the new issue #669 I just added hints to, I think there are still a few corner cases in the new read interface that need to be sorted out. In addition to that issue which I think is a true regression, the behavior surrounding variable-length string arrays is maybe now a bit confused —

writing on v0.13.6 — while not truly round-trippable — were at least similar in form for read and write:

julia> h5open("test.h5", "w") do hfile
           attrs(hfile)["attr"] = HDF5.HDF5Vlen(["string", "another"])
       end
HDF5Vlen{HDF5.ASCIIChar}(["string", "another"])

julia> h5open("test.h5", "r") do hfile
           read(attrs(hfile)["attr"])
       end
2-element Vector{String}:
 "string"
 "another"

but now on master, the form being read back is very different from that being written:

julia> h5open("test.h5", "w") do hfile
           attrs(hfile)["attr"] = HDF5.HDF5Vlen(["string", "another"])
       end
HDF5Vlen{HDF5.ASCIIChar}(["string", "another"])

julia> h5open("test.h5", "r") do hfile
           read(attrs(hfile)["attr"])
       end
2-element Vector{Vector{String}}:
 ["s", "t", "r", "i", "n", "g"]
 ["a", "n", "o", "t", "h", "e", "r"]

I haven't dug far enough yet to know the answer for sure, but I think this is due to special cases for HDF5Vlen strings that construct the "strange" length-1 string arrays that are plaguing MAT.jl right now on HDF5 master.

Maybe all that is just to say, with the large overhaul of the read interface, there's probably need to look at the write interface as well.

Edit: changed a wrong "fixed" to "variable" in the intro which I'd gotten backwards

[kleinhenz]

Yeah I agree the write side could be cleaned up somewhat also. Just to note, your example roundtrips fine if you remove the HDF5Vlen from the write. As far as I can tell the only purpose of HDF5Vlen is to write these strange string types. It is mostly possible to tell from the type itself whether it should be written as a vlen type so HDF5Vlen doesn't seem to serve a clear purpose and I think it should probably be deprecated/removed.

[jmert]

Just to note, your example roundtrips fine if you remove the HDF5Vlen from the write.

Yeah, I purpose-found this case in trying to track down how MAT.jl currently writes the DATATYPE H5T_VLEN { H5T_STRING { STRSIZE 1; ...}} HDF5 type attributes.

...so HDF5Vlen doesn't seem to serve a clear purpose and I think it should probably be deprecated/removed.

If this type were removed, what would be the call that a user like MAT.jl would have to use to force writing out a variable-length string rather than a fixed-length string? Would it be to create the corresponding HDF5 data type manually instead of just using a pre-existing datatype() return?

[kleinhenz]

HDF5Vlen currently doesn't write variable length strings but rather variable length arrays of length 1 fixed strings. This seems to be a particular requirement of the MAT format so I would move the code for writing this format into MAT.jl which would involve defining the datatype and using some pieces of the lower level interface. I think this is probably fine since this shouldn't be visible to MAT users.

[kleinhenz]

I can try to make a pull request to fix MAT to see how it would work.

[jmert]

HDF5Vlen currently doesn't write variable length strings but rather variable length arrays of length 1 fixed strings. This seems to be a particular requirement of the MAT format ...

OK, I think I'm starting to follow this more clearly. Yes, empirically, it's what MATLAB does with its data format, but it's also (maybe unfortunately) the "natural" result of naively constructing a variable-length array of HDF5 strings:

julia> dtype1 = HDF5.HDF5Datatype(HDF5.h5t_vlen_create(HDF5.h5t_copy(HDF5.H5T_C_S1)))
HDF5 datatype: H5T_VLEN {
      H5T_STRING {
         STRSIZE 1;
         STRPAD H5T_STR_NULLTERM;
         CSET H5T_CSET_ASCII;
         CTYPE H5T_C_S1;
      }
   }

Instead, it looks like what looks more "sane" from an outside perspective is to first manipulate the string type to be variable length and then construct the variable array around them:

julia> dvstr = HDF5.h5t_copy(HDF5.H5T_C_S1);
julia> HDF5.h5t_set_size(dvstr, HDF5.H5T_VARIABLE);
julia> dtypeN = HDF5.HDF5Datatype(HDF5.h5t_vlen_create(dvstr))
HDF5 datatype: H5T_VLEN {
      H5T_STRING {
         STRSIZE H5T_VARIABLE;
         STRPAD H5T_STR_NULLTERM;
         CSET H5T_CSET_ASCII;
         CTYPE H5T_C_S1;
      }
   }

It seems like there might be some motivation to keep around the HDF5Vlen data type to encapsulate the H5T_VLEN type, but I also agree that the current behavior where HDF5Vlen of strings is translated into the former HDF5 datatype is probably something that should be done manually by MAT.jl so that the default behavior here is something more like the latter.

(For reference, I eventually found some hints to his at https://support.hdfgroup.org/HDF5/doc/RM/RM_H5T.html#CreateVLString)

[musm]

Should we plan for a new release soon? Anything in particular we should get in beforehand? We have made quite a few changes since 0.13, so it feels like a good time. I know there's still lots left to be done, but I wonder if it would be better to make incremental major releases or one large major release that would require many user changes to update to. I'm kind of leaning towards the former, but don't have strong opinions on it.

[jmert]

I'd like to see the type-specific overloads of h5a_* and h5d_* turned into specializations of a_* and d_* before the actual release, since those are arguably part of the middle-level layer and shouldn't be mixed in with the true low-level interface functions.

I'll try to work on that over the next day or two. I think doing just the mechanical translation should be relatively fast and straightfoward. There's probably more reorganizing and cleanup that could be done once the renames happen, but that can happen later in point releases.

[musm]

I think we are at the stage where it makes sense to release 0.14. There was some follow up work in upgrading the underlying hdf5 library to 0.12 that I am working on, but it's quite tricky since HDF5 doesn't allow for cross compilation.

[musm]

Interesting issue number