Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

More memory needed for quilting nodes to run version 41b1724 #2008

Open
hu5970 opened this issue Nov 22, 2023 · 1 comment
Open

More memory needed for quilting nodes to run version 41b1724 #2008

hu5970 opened this issue Nov 22, 2023 · 1 comment

Comments

@hu5970
Copy link

hu5970 commented Nov 22, 2023

The model in RRFS NA 3km real-time parallel with the recent update (Hash 41b1724) crashed during first several minutes of the forecast. Further investigation shows the crash is because of out-of-memory issue from a core for quilting output files. After double the number cores used for quilting from 64 to 128, the forecast runs smoothly.

@junwang-noaa
Copy link
Collaborator

@hu5970 There were recent UPP updates on write grid comp which requires more memory. Please see the comments in PR#1946:

#1946 (comment)

To confirm, would you please run a test with ufs-weather-model revision 9e41ad9 (before #1946 was committed) to see if your test can pass without increasing the write tasks? Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants