Port to Intel LLVM on Hera, Orion and Hercules

[GeorgeGayno-NOAA]

DESCRIPTION OF CHANGES:

Port to Intel LLVM on Hera, Orion and Hercules. Keep the Intel Classic as an option on those machines.

TESTS CONDUCTED:

If there are changes to the build or source code, the tests below must be conducted. Contact a repository manager if you need assistance.

• Compile branch on all Tier 1 machines using Intel Classic (Orion, Jet, Hera, Hercules and WCOSS2). Done c868685.
• Compile branch on Orion, Hera and Hercules using Intel LLVM. Done using c868685.
• Compile branch on Hera using GNU. Done using c868685.
• Compile branch in 'Debug' mode on WCOSS2. Done using c868685. No warnings.
• Run unit tests locally on any Tier 1 machine. Done on Hera using c868685 (Intel LLVM).
• Run relevant consistency tests locally on all Tier 1 machines. See below.

• The consistency tests passed on Jet and WCOSS2 as expected (still using Intel Classic). See: comment.
• The cpld_gridgen tests failed on Hera, Hercules and Orion. But differences were insignificant. See: comment and comment.
• The ocnice_prep tests failed on Hera. But differences were insignificant. See: comment.
• Some of the chgres_cube tests failed on Hera, Orion and Hercules. But differences were insignificant. See: comment.
• Some of the grid_gen tests failed on Hera, Orion and Hercules, but differences were insignificant. The global_cycle, ice_blend, snow2mdl and weight_gen tests passed. See: comment and comment.

Additional tests performed. Those on Hera, Orion and Hercules used Intel LLVM.

• Test gdas_init utility on Hera. Successfully done using c868685 with a 2024/11/01/06z GFS test case.
• Test sfc_climo_gen utility on Hera and WCOSS2. Successfully done using c868685.
• Test weight_gen utility on Hera, Orion and Hercules. Successfully tested using c868685.
• Create a global C96 model grid using the scripts in ./driver_scripts on Hera, Orion and Hercules. Successfully done using 7efdbed.

DEPENDENCIES:

None.

DOCUMENTATION:

N/A

ISSUE:

Fixes #879.

[GeorgeGayno-NOAA]

All consistency tests were run on Hera using 7efdbed and Intel LLVM.

• The global_cycle, ice_blend, snow2mdl and weight_gen tests all passed.
• chgres_cube tests 1, 2, 4, 5, 6, 7, 8, 9, 10, 11 and 12 failed. Differences from the baseline were 'roundoff' in magnitude.
• cpld_griden and ocnice_prep failed. Will contact Denise for review.
• grid_gen tests 1, 2, 3, 6, 7. Differences from the baseline were 'roundoff' in magnitude.

[GeorgeGayno-NOAA]

All consistency tests were run on Hercules and Orion using 7efdbed and Intel LLVM. Results were the same on both machines.

• The ice_blend, global_cycle, ocnice_prep, snow2mdl and weight_gen tests all passed.
• grid_gen tests 1, 2, 3, 6 and 7 failed.
• chgres_cube tests 6, 7 and 11 failed.

[GeorgeGayno-NOAA]

The cpld_gridgen regression tests were run using the branch at 2acab86 (after cpld_gridgen mods were merged in from 'develop'). The tests failed on Hera, Orion and Hercules. Will ask @DeniseWorthen to check if the differences are a problem. The log files are here:

• Hera: /scratch1/NCEPDEV/da/George.Gayno/ufs_utils.git/UFS_UTILS/reg_tests/cpld_gridgen/for.denise
• Orion: /work/noaa/da/ggayno/save/ufs_utils.git/UFS_UTILS/reg_tests/cpld_gridgen/for.denise
• Hercules: /work2/noaa/da/ggayno/save/UFS_UTILS/reg_tests/cpld_gridgen/for.denise

[DeniseWorthen]

@GeorgeGayno-NOAA I'm not able to see directories in the da project space, but I was able to look at the Hera logs. These all look like round-off level differences. I think this is all OK (assuming the hercules and orion diffs show the sort of differences).

[GeorgeGayno-NOAA]

@GeorgeGayno-NOAA I'm not able to see directories in the da project space, but I was able to look at the Hera logs. These all look like round-off level differences. I think this is all OK (assuming the hercules and orion diffs show the sort of differences).

Thanks @DeniseWorthen, far fewer records are different on Hercules and Orion, and the differences are 'round-off' in magnitude.

[GeorgeGayno-NOAA]

All regression tests were run on Jet and WCOSS2 using c868685. Since UFS_UTILS still uses Intel Classic on these machines, all tests passed as expected.

[GeorgeGayno-NOAA]

The chgres_cube regression tests were rerun using c868685, which merged in changes to that program from 'develop'. Some tests failed:

• Orion and Hercules - Tests 6, 7, and 11.
• Hera - Tests 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12 and 14

The output from these tests was examined. All differences from the baseline were very small (considered 'round-off'). The IntelLLVM changes are working.

[GeorgeGayno-NOAA]

The ocnice_prep regression tests were run on Hera, Orion and Hercules using c868685.

The tests passed on Hercules and Orion.

The tests failed on Hera. However, the differences with the baseline were very small - on the order of 'floating point'. The IntelLLVM changes are working.

[GeorgeGayno-NOAA]

@DusanJovic-NOAA - would you mind doing a quick review?

[DusanJovic-NOAA]

cmake and module changes look good. I'm not familiar with spack configurations, and shell scripts, but if all tests pass I think this PR can be merged.

[DavidHuber-NOAA]

Changes look good. I will open a follow-on issue in the global workflow to incorporate this. For now, we will need to use the classic compilers until the GSI, GDASApp, and the GFS Utils are able to compile with OneAPI compilers. Thanks for the work on this!

[DavidHuber-NOAA]

Opened issue NOAA-EMC/global-workflow#3099.