need a flag to indicate netcdf files being written completely

[jiandewang]

Is your feature request related to a problem? Please describe.
in UFS coupled system, post processing job rely on a flag which indicates that the forecast has finished at certain time length (used as input file for post job) to be launched. In MOM6 output files start to appear on disk (with garbage data) before the forecast length has been reached.

Describe the solution you'd like
need a flag file to be generated at the end when certain forecast length has reached and output files are written completely, so that post processing job can be trigged and read in real data.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[jiandewang]

FYI: EMC internal discussion NOAA-EMC/global-workflow#935

[rem1776]

@jiandewang Do you happen to know which version of the IO is being used by MOM6 when this occurs? There's two version used in the mom6 repo, FMS2 has wrappers for our updated io (fms2io) while FMS1 uses the older routines (mpp_io) that won't be updated.

https://github.com/mom-ocean/MOM6/tree/main/config_src/infra

I guess this is coming from netcdf files that are opened and getting written to but haven't been closed yet. We could add an option to the close fle routine to write the flag file when the netcdf file gets closed, would that potentially work for this? Alternatively this could be done from MOM6 as well just after the close call, doesn't necessarily need to be from FMS.

[uramirez8707]

@jiandewang Is this needed for history files (files generated via the diag_manager) or RESTART files?

[bensonr]

@jiandewang - This issue has been discussed internally and the decision has been made to deny this enhancement request. If you are using the frequency cycling capabilities of the FMS diag_manager via the file specification in the diag_table, you are aware of the simulation datestamps for cycling to a new history/diagnostic file (closing previous one and opening appropriately datestamped new one). As the close will trigger a flush of the buffered NetCDF data to the file and filesystem, the post-processing can be triggered on the existence of the next file in the sequence. Alternately, you should be able to trigger post from simulation datestamps in the simulation stdout.

With these alternate approaches, and the high potential for a separate marine write-component for the UFS, this request is being marked as invalid and wontfix. Feel free to reopen if you feel this needs more discussion.

[jiandewang]

@bensonr thanks for the heads up.