Make Darshan work

[lyon-fnal]

Try Darshan with Julia. It loads with LD_PRELOAD. May need to change HDF5.jl and MPI.jl so that they don't explicitly give the library name on ccall`.

[lyon-fnal]

Problem in MPI.jl. The shared object needs to be loaded with Libdl.dlopen in MPI.__init__ because loading the .so is a runtime thing. But in src/MPI.jl close to the top there's an include("implementations.jl") That defines Get_library_version which does a ccall to MPI_Get_library_version. That would be ok, except that in that file, it calls Get_library_version(). Since this happens before MPI.__init__, the library isn't loaded and it fails. Move this call to the __init__ function and make MPI_LIBRARY_VERSION_STRING a reference. Do the same thing for Get_version too further down in implementations.jl.

In fact I think all of implementations.jl (or at least all of the calls) needs to be moved to the __init__ function.'

Actually - scratch the work with implementations.jl - all it does is get versions and implementation types. So it's fine to do the ccall with the library - that stuff should not be overridden.

[lyon-fnal]

Darshan works on NERSC Cori Haswell! Here's a summary of what I did...

• I have my own build of Darshan 3.2.1 against cray-hdf5-parallel/1.12.0.0 and cray-mpich/7.7.10 (see here for configure and build instructions). Configure has --enable-hdf5-mod=$HDF5_DIR.
• The Darshan build includes a module specification. Activate it with,

module rm darshan   # Unload the default Cori Darshan (currently an old 3.1.7)
module use $HOME/apps.cori-hsw/darshan-3.2.1/share/craype-2.x/modulefiles
module load darshan

• Build my modified MPI.jl and HDF5.jl against system libraries (see above).
• Run a simple script with srun --export=ALL,LD_PRELOAD=libdarshan.so julia --project tryit.jl (need to run in batch or on a Cori node with sallocate).

Things to do:

• Submit PRs for MPI.jl #450 and HDF5.jl #791
• Try with a more complex script
• Try with the package compiler (I don't think that will make a difference)

[lyon-fnal]

So I have this working with HDF5.jl. I've altered energyByCal.jl to make collective reads an option. Even with it turned on, I don't see any collective reads in Darshan (MPIIO_COLL_READS is 0). Even with H5D_USE_MPIIO_COLLECTIVE true for each column. Not sure what's going on there. Seems like HDF5 is making some decision about using collective reads or not.

mpio.jl DOES do collective reads. For that, each rank reads from its own chunk.

Trying mpio2.jl where ranks will not all read from their own chunk.

[lyon-fnal]

mpio2.jl (which is now a nice example of collective writes) does read many chunks. And it's doing a collective read (one per rank). Not sure why energyByCal.jl does no collective reads. Maybe this doesn't really matter.