Fearture/jjin3/allow for missing ob type b

[gmao-jjin3]

Allow filter for missing satellite radiance files in the final SWELL configurations, following @rtodling PR #400. However, radiance observations with zero-sized input files are omitted in the final configurations because JEDI/IODA cannot read those zero-sized files. In addition, filters for missing conventional and ozone observational files are omitted since an action of "error" is now assigned for those missing observations.

[gmao-jjin3]

So far, this PR works for missing convectional, ozone, and microwave radiance data. It crashes swell-3dfgat_atmos run when any of AIRS, IASI, CRIS-FSR, and AVHRR input files is missing.

[gmao-jjin3]

@gmao-wgu It turns out that JEDI crashes with this correlated IR observational error inflation filter when there aren't observational input files:

- filter: BlackList
  filter variables:
  - name: brightnessTemperature
    channels: *cris-fsr_n20_channels
  action:
    name: inflate error
    inflation variable:
      name: ObsFunction/ObsErrorFactorWavenumIR
      channels: *cris-fsr_n20_channels
      options:
        channels: *cris-fsr_n20_channels

UserError: Invalid prior filter requested
ObsErrorCrossVarCov: Obs error correlations not provided for 324 of 431 channels/variables in file: fv3-jedi/rcov/cris-fsr_108_jedi_rcov.nc4. To see which channels, turn on OOPS_TRACE
OOPS Ending 2024-09-03 12:25:59 (UTC-0400)

I added a check of number of observations in /discover/nobackup/jjin3/jedi/jedi-work2/ufo/src/ufo/errors/ObsErrorCrossVarCov.cc.jj . However, that doesn't solve the issue. JEDI still returns an error of "Invalid prior filter". Any idea?

[gmao-wgu]

@gmao-jjin3 I will try to replicate your case and proceed from there.

[gmao-jjin3]

@gmao-wgu It turns out there is an issue to handle IASI files in JEDI/ioda. The filter mentioned above is fine. My other tests go through with missing AIRS and CRIS-FSR input observational files. Tests only fail with missing IASI data. UFO filters are fine.
Reported the issue to the JCSDA team: https://github.com/JCSDA-internal/ioda/issues/1332

[gmao-jjin3]

It turns out I had a minor update in UFO before I traced down the above issue in IODA. Here is a UFO PR for the update: https://github.com/JCSDA-internal/ufo/pull/3455.

[Dooruk]

@gmao-jjin3 and @rtodling, is this ready to merge? I see the relevant PR in JEDI is merged.

[gmao-jjin3]

@Dooruk No, the JEDI team is still looking into the issue that arises with missing IASI data.

[fabiolrdiniz]

@gmao-jjin3, please notice that the actual error message in your excerpt above is:

UserError: Invalid prior filter requested

which means you are probably adding a filter in the wrong section (pre, prior, post, please see https://github.com/JCSDA-internal/oops/blob/d77217323bbbd02ea41049fe1bd339f24a64f177/src/oops/base/ObsFilters.h#L41-L58 for more info), while

ObsErrorCrossVarCov: Obs error correlations not provided for 324 of 431 channels/variables in file: fv3-jedi/rcov/cris-fsr_108_jedi_rcov.nc4. To see which channels, turn on OOPS_TRACE
OOPS Ending 2024-09-03 12:25:59 (UTC-0400)

is a warning message that the matrix you provided for CrIS-FSR doesn't have covariances for some channels (likely the ones you don't use).

[gmao-jjin3]

@fabiolrdiniz I noticed that before. However, I don't think it is the real reason. All the filters and their order are fine when the obs input file is available.

[Dooruk]

@fabiolrdiniz or @jianjunj, as I recall, there was an issue associated with the UFO filters in case the missing file action was used. Was that resolved? Can we go ahead and merge these changes?

[fabiolrdiniz]

@fabiolrdiniz or @jianjunj, as I recall, there was an issue associated with the UFO filters in case the missing file action was used. Was that resolved? Can we go ahead and merge these changes?

That's what I recall too and this was the associated fix https://github.com/JCSDA-internal/ufo/pull/3455. However, I remember @gmao-jjin3 reported elsewhere he was still having an issue that I couldn't reproduce on my side with fv3-jedi (gfs though). Same observation config (obs space, files, filters) but with different settings for the solver. I don't think the solver differences would play a role though.

I'm not the right person to say it's working on the swell environment, but it would be nice if someone could test it and report if indeed there is still an issue.

[Dooruk]

That's what I recall too and this was the associated fix JCSDA-internal/ufo#3455. However, I remember @gmao-jjin3 reported elsewhere he was still having an issue that I couldn't reproduce on my side with fv3-jedi (gfs though). Same observation config (obs space, files, filters) but with different settings for the solver. I don't think the solver differences would play a role though.

I'm not the right person to say it's working on the swell environment, but it would be nice if someone could test it and report if indeed there is still an issue.

@gmao-wgu or @gmao-yzhu you mentioned this issue was resolved but could you point the PR or Git Issue that resolved this issue before we merge this in? And @jianjunj if possible.

[gmao-yzhu]

Thanks for the reminder. Let me contact Jianjun and I’ll get back to you later. From: Doruk Ardağ ***@***.***> Date: Tuesday, November 12, 2024 at 10:29 AM To: GEOS-ESM/swell ***@***.***> Cc: Zhu, Yanqiu (GSFC-6101) ***@***.***>, Mention ***@***.***> Subject: [EXTERNAL] [BULK] Re: [GEOS-ESM/swell] Fearture/jjin3/allow for missing ob type b (PR #415) CAUTION: This email originated from outside of NASA. Please take care when clicking links or opening attachments. Use the "Report Message" button to report suspicious messages to the NASA SOC. That's what I recall too and this was the associated fix JCSDA-internal/ufo#3455<JCSDA-internal/ufo#3455>. However, I remember @gmao-jjin3<https://github.com/gmao-jjin3> reported elsewhere he was still having an issue that I couldn't reproduce on my side with fv3-jedi (gfs though). Same observation config (obs space, files, filters) but with different settings for the solver. I don't think the solver differences would play a role though. I'm not the right person to say it's working on the swell environment, but it would be nice if someone could test it and report if indeed there is still an issue. @gmao-wgu<https://github.com/gmao-wgu> or @gmao-yzhu<https://github.com/gmao-yzhu> you mentioned this issue was resolved but could you point the PR or Git Issue that resolved this issue before we merge this in? And @jianjunj<https://github.com/jianjunj> if possible. — Reply to this email directly, view it on GitHub<#415 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ARUMEBXDGRON4IDQXMS5RXT2AINHDAVCNFSM6AAAAABNLH4IK6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINZQHA2DGNBWGI>. You are receiving this because you were mentioned.Message ID: ***@***.***>