NCIO bug on Hercules

[DavidHuber-NOAA]

Describe the bug
When creating a dataset via spack-stack/1.5.1 NCIO, a netCDF/HDF error is generated. However, when the NCIO library is installed standalone on Hercules, there is no issue.

To Reproduce
I will build a sample test case and update this section. For now, you can see the failed, spack-stack log file here: /work/noaa/global/dhuber/para/com/cycled_herc/logs/2021110900/gdasanalcalc.spack-stack.log and the successful log file (built with a standalone version of NCIO, using spack-stack netcdf-c and netcdf-fortran libraries and the same Intel compilers) here: /work/noaa/global/dhuber/para/com/cycled_herc/logs/2021110900/gdasanalcalc.standalone.log.

The failure occurs on the call of calc_anl.x at line 208 of gsi_utils/netcdf_io/calc_analysis/init_io.f90.

The standalone build of NCIO is available in /work/noaa/global/dhuber/LIBS/NCIO/ncio_112/install and the source code is in /work/noaa/global/dhuber/nceplibs-ncio. It was built following this recipe:

git clone [email protected]:NOAA-EMC/nceplibs-ncio
cd nceplibs-ncio
git checkout tags/v1.1.2 -b 1.1.2
module purge
module use /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.5.1/envs/gsi-addon/install/modulefiles/Core
module load stack-intel
module load stack-intel-oneapi-mpi
module load netcdf-c
module load netcdf-fortran
module load cmake
mkdir build
cd build
cmake .. --DCMAKE_INSTALL_PREFIX=/work/noaa/global/dhuber/LIBS/NCIO/ncio_112/install
make -j 2
make install

Expected behavior
The NCIO library would be able to create a dataset.

System:
What system(s) are you running the code on?
Hercules

Additional context
Found while cycle-testing the global workflow on Hercules (NOAA-EMC/global-workflow#1588).

[DavidHuber-NOAA]

I have built a test case here: /work/noaa/global/dhuber/LIBS/NCIO/analcalc_test_case/calcanl_ensres_06. In doing so, I found that this is an intermittent issue, so the standalone version of NCIO probably doesn't actually fix this, it just got lucky. I will investigate this further.

[climbfuji]

I have built a test case here: /work/noaa/global/dhuber/LIBS/NCIO/analcalc_test_case/calcanl_ensres_06. In doing so, I found that this is an intermittent issue, so the standalone version of NCIO probably doesn't actually fix this, it just got lucky. I will investigate this further.

This sounds a lot like the intermittent netCDF problems that the ufs-weather-model folks have seen and are trying to figure out.

[AlexanderRichert-NOAA]

We ran into netcdf issues when running on Lustre. Can you test under non-LFS space, e.g., $HOME? If so then yeah same issue.

[DavidHuber-NOAA]

Sure enough, it works on $HOME. I'll go ahead and close this and follow the netCDF problem the ufs is working on. Thanks!