Bufr2IODA working in conjunction with Var

[rtodling]

Description

This is preliminary work to introduce and test extra configuration to add pre-filters needed for intrustruments whose IODA files have been generated from bufr files as opposed to from NCdiags.

Still pending:

• handle for files beyond IASI and AMSUA instruments
• addiition in swell to be able to append pre-filters to observation yamls (notice this can be placed at the end of the obs space for instrument.
• change workflows of Var to allow for opt to pull bufr: (i) get bufr files from archive; and (ii) convert to IODA. This flow already exists in what @cohen-seth has done so far.

NOTICE NOTICE NOTICE: In the results below, I am not too concerned on how things compare w/ GSI and how correct or not the increments are. All I am concerned about is how the results of develop comare w/ results when the ncdiag-generated IODA files are replaced with files generated from the bufr converter.

Dependencies

• waiting on Entries needed when using IODA that comes from bfr converter #673

Impact

For now: None

[rtodling]

Results for AMSU-A N19

[rtodling]

Results for IASI Metop-B

[Dooruk]

Sorry, this is coming from a limited understanding space and I would like to grasp the approach better. My understanding was that most of what GSI does (thinning/filtering) would be tackled by ObsBuilder, which is build within bufr-query.

However does this draft PR indicate that for certain instruments we won't even need the ObsBuilder package and can just let JEDI handle GSI level filtering that was not necessary while we were using ncdiag files?

[rtodling]

Sorry, this is coming from a limited understanding space and I would like to grasp the approach better. My understanding was that most of what GSI does (thinning/filtering) would be tackled by ObsBuilder, which is build within bufr-query.

However does this draft PR indicate that for certain instruments we won't even need the ObsBuilder package and can just let JEDI handle GSI level filtering that was not necessary while we were using ncdiag files?

No, I don't think so. In my opinion the ioda file should be as plain as possible; As you say, GSI does all the thinning and filters, and analogously it should be for JEDI-var to do thinning. Otherwise we cannot have a database of bufr-coverted IODA files that are suitable for general experiments; in order works, we don't want IODA files that configured to particular choices of thinning and what-no, which when revised in the future would force us to re-create the database of IODA files.

You are right that IODA files generated from NCdiags are already thinning in a particular way, but that's because when the output of GSI was designed to feed into JEDI, the "interface" was created at a low level; whereas it could have been created at a higher level, right after GSI reads the bufr input data.