Clone, build, and run C48_ATM, C48_S2SW, and C96_atm3DVar on Gaea C5 and C6

[DavidBurrows-NCO]

Description

What:
Correct build/run for C48_ATM, C48_S2SW, and C96_atm3DVar on Gaea C5. Add build and run capability for C48_ATM, C48_S2SW, and C96_atm3DVar on Gaea C6.
Why:
After the C5 OS upgrade, submodules no longer built in the global-workflow. This PR correct that and adds build/run capability to C6.

Resolves #3011
Refs ufs-wx-model 2448
Refs ufs-utils 995
Refs gfs-utils 87
Refs upp 1070
Refs gsi-enkf 800
Refs gsi-utils 55
Regs gsi-monitor 146
Refs GDAS 1361

Type of change

• Bug fix (fixes something broken)
• New feature (adds functionality)

Change characteristics

• Is this a breaking change (a change in existing functionality)? NO
• Does this change require a documentation update? NO
• Does this change require an update to any of the following submodules? YES (If YES, please add a link to any PRs that are pending.)
  • EMC verif-global
  • GDAS
  • GFS-utils NOAA-EMC/gfs-utils87
  • GSI
  • GSI-monitor
  • GSI-utils
  • UFS-utils Update scripts to build on both Gaea-C5 and C6 ufs-community/UFS_UTILS#995
  • UFS-weather-model Gaea C6 support for UFSWM ufs-community/ufs-weather-model#2448
  • wxflow

How has this been tested?

C5 and C6: clone, built, and ran C48_ATM and C48_S2SW successfully.
C96_atm3DVar is hanging in sfcanl jobs.

Checklist

• Any dependent changes have been merged and published
• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have documented my code, including function, input, and output descriptions
• My changes generate no new warnings
• New and existing tests pass with my changes
• This change is covered by an existing CI test or a new one has been added
• Any new scripts have been added to the .github/CODEOWNERS file with owners
• I have made corresponding changes to the system documentation if necessary

[DavidBurrows-NCO]

Hi @aerorahul @WalterKolczynski-NOAA We're still waiting on build merges for some submodules, so I've left this PR in draft. From our conversation Tuesday, I've pointed the submodules that were merged to their respective head of develop and the others to my commit for now. Should I be pointing to my submodule commits instead to limit the number of changes coming into GW? Thanks

[jswhit]

sorc/build_all.sh needs the following update:

--- sorc/build_all.sh
+++ sorc/build_all.sh
@@ -149,7 +149,7 @@ build_opts["ww3prepost"]="${_wave_opt} ${_verbose_opt} ${_build_ufs_opt} ${_buil

 # Optional DA builds
 if [[ "${_build_ufsda}" == "YES" ]]; then
-   if [[ "${MACHINE_ID}" != "orion" && "${MACHINE_ID}" != "hera" && "${MACHINE_ID}" != "hercules" && "${MACHINE_ID}" != "wcoss2" && "${MACHINE_ID}" != "noaacloud" && "${MACHINE_ID}" != "gaea" ]]; then
+   if [[ "${MACHINE_ID}" != "orion" && "${MACHINE_ID}" != "hera" && "${MACHINE_ID}" != "hercules" && "${MACHINE_ID}" != "wcoss2" && "${MACHINE_ID}" != "noaacloud" && "${MACHINE_ID}" != "gaeac5" && "${MACHINE_ID}" != "gaeac6" ]]; then
       echo "NOTE: The GDAS App is not supported on ${MACHINE_ID}.  Disabling build."
    else
       build_jobs["gdas"]=8

[jswhit]

also ush/load_ufsda_modules.sh needs

--- a/ush/load_ufsda_modules.sh
+++ b/ush/load_ufsda_modules.sh
@@ -34,13 +34,13 @@ source "${HOMEgfs}/ush/module-setup.sh"
 module use "${HOMEgfs}/sorc/gdas.cd/modulefiles"

 case "${MACHINE_ID}" in
-  ("hera" | "orion" | "hercules" | "wcoss2")
+  ("hera" | "orion" | "hercules" | "gaeac5" | "gaeac6" | "wcoss2")
     module load "${MODS}/${MACHINE_ID}"
     ncdump=$( command -v ncdump )
     NETCDF=$( echo "${ncdump}" | cut -d " " -f 3 )
     export NETCDF
     ;;
-  ("jet" | "gaea" | "s4" | "acorn")
+  ("jet" | "s4" | "acorn")
     echo WARNING: UFSDA NOT SUPPORTED ON THIS PLATFORM
     ;;
   *)

[DavidBurrows-NCO]

also ush/load_ufsda_modules.sh needs

Thanks @jswhit I pushed changes to ush/load_ufsda_modules.sh and sorc/build_all.sh

[jswhit]

also...

workflow/hosts/gaeac6.yaml and gaeac5.yaml:

-QUEUE_SERVICE: normal
+QUEUE_SERVICE: hpss
 PARTITION_BATCH: batch
-PARTITION_SERVICE: batch
+PARTITION_SERVICE: dtn_f5_f6

and modulefiles/module_gwsetup.gaeac6.lua:

-prepend_path("MODULEPATH", "/ncrc/proj/epic/spack-stack/spack-stack-1.6.0/envs/unified-env/install/modulefiles/Core")
+prepend_path("MODULEPATH", "/ncrc/proj/epic/spack-stack/c6/spack-stack-1.6.0/envs/unified-env/install/modulefiles/Core")

[jswhit]

env/GAEAC5.env and GAEAC6.env seem to be missing a bunch of stuff. I just copied HERCULES.env for both, and made some minor mods (see https://github.com/jswhit2/global-workflow/blob/develop/env/GAEAC5.env)

[jswhit]

build_ww3prepost is failing for me on both c5 and c6 (using ufs-wx-model 2448)

[DavidBurrows-NCO]

missing a bunch of stuff

@jswhit It's not really missing but intentionally minimized at the request of EMC porting to a new machine. Instead, we started from a nearly blank canvas and have been building up. Currently, the C5 and C6.env files are set up for C48_ATM, C48_S2SW, and C96_atm3DVar jobs. The 3DVarAOWCDA configuration you're running will definitely have some additional jobs. If you send those particular job names (or "step" in the env file). I will add them to the files.

[JessicaMeixner-NOAA]

build_ww3prepost is failing for me on both c5 and c6 (using ufs-wx-model 2448)

@jswhit - can you point me to a log file? Maybe I can look and see if something is easy to fix with this.

[jswhit]

build_ww3prepost is failing for me on both c5 and c6 (using ufs-wx-model 2448)

@jswhit - can you point me to a log file? Maybe I can look and see if something is easy to fix with this.

@JessicaMeixner-NOAA here is the error:

gpfs/f6/ira-da/proj-shared/Jeffrey.S.Whitaker/global-workflow-jswhit2/sorc/ufs_model.fd/WW3/model/src/w3initmd.F90(451): error #7002: Error in opening the compiled module file.  Check INCLUDE paths.   [WAV_RESTART_MOD]    use wav_restart_mod, only : read_restart
--------^
/gpfs/f6/ira-da/proj-shared/Jeffrey.S.Whitaker/global-workflow-jswhit2/sorc/ufs_model.fd/WW3/model/src/w3initmd.F90(975): error #6632: Keyword arguments are invalid without an explicit interface.   [VA]            call read_restart(trim(fname), va=va, mapsta=mapsta, mapst2=mapst2)
-------------------------------------------^
/gpfs/f6/ira-da/proj-shared/Jeffrey.S.Whitaker/global-workflow-jswhit2/sorc/ufs_model.fd/WW3/model/src/w3initmd.F90(975): error #6632: Keyword arguments are invalid without an explicit interface.   [MAPSTA]
            call read_restart(trim(fname), va=va, mapsta=mapsta, mapst2=mapst2)
--------------------------------------------------^
/gpfs/f6/ira-da/proj-shared/Jeffrey.S.Whitaker/global-workflow-jswhit2/sorc/ufs_model.fd/WW3/model/src/w3initmd.F90(975): error #6632: Keyword arguments are invalid without an explicit interface.   [MAPST2]
            call read_restart(trim(fname), va=va, mapsta=mapsta, mapst2=mapst2)
-----------------------------------------------------------------^
/gpfs/f6/ira-da/proj-shared/Jeffrey.S.Whitaker/global-workflow-jswhit2/sorc/ufs_model.fd/WW3/model/src/w3initmd.F90(451): error #6580: Name in only-list does not exist or is not accessible.   [READ_RESTART]
    use wav_restart_mod, only : read_restart
--------------------------------^
compilation aborted for /gpfs/f6/ira-da/proj-shared/Jeffrey.S.Whitaker/global-workflow-jswhit2/sorc/ufs_model.fd/WW3/model/src/w3initmd.F90 (code 1)

[JessicaMeixner-NOAA]

@jswhit - Okay I know what the issue is, but it'll take a minute to get it fixed. The issue crept in with ufs-community/ufs-weather-model#2445 and we didn't catch it. If you go back one-commit of ufs-waether-model, hopefully things will run. We'll get a fix in as soon as possible.