Add 3DVar atmosphere regression test

[CoryMartin-NOAA]

Description

This PR adds the framework for an atmospheric C384 3DVar regression test. In the short term, this can be used to turn on additional observation types for @RussTreadon-NOAA 's cycling prototype but longer term can serve as the basis for a more rigorous, realistic regression testing system.

This is in draft until

• input data are staged on supported platforms
• compute resource requirements are fleshed out for supported platforms

But it works on Ursa and would appreciate any general feedback so far.

Companion PRs

None

Issues

Closes #1953

Automated CI tests to run in Global Workflow

• atm_jjob
• C96C48_ufs_hybatmDA
• C96C48_hybatmsnowDA
• C96_gcafs_cycled
• C48mx500_3DVarAOWCDA
• C48mx500_hybAOWCDA
• C96C48_hybatmDA

[Copilot]

Pull Request Overview

This PR adds a framework for running atmospheric C384 3DVar regression tests. The implementation provides a complete test harness that can stage input data, generate job scripts for different schedulers, and execute the 3DVar analysis on supported HPC platforms.

Key Changes:

• Introduces a new regression test script for atmospheric 3DVar analysis with C384 resolution
• Adds a Python utility to generate job submission scripts for both SLURM and PBS schedulers
• Configures CMake to conditionally enable regression tests when test data is available

Reviewed Changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
test/generate_job_script.py	New Python script that generates scheduler-specific job submission scripts from YAML configuration
test/atm/regression_test_3dvar.sh	Main regression test script that sets up and executes the atmospheric 3DVar test
test/atm/CMakeLists.txt	Adds CMake test target for the 3DVar regression test
test/CMakeLists.txt	Implements conditional logic to enable regression tests based on data availability
modulefiles/GDAS/ursa.intel.lua	Configures the regression test data path for the Ursa platform

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[CoryMartin-NOAA]

This will remain in draft until input data can be copied to all supported platforms but that should wait until any issues with NOAA-EMC/JEDI-T2O#147 are resolved and the file naming conventions are agreed to

[DavidNew-NOAA]

Why need to do so much manually in regression_test_3dvar.py? Why can't you use existing infrastructure in GW and GDASApp like the AtmAnalysis class and jcb_base.yaml.j2?

[CoryMartin-NOAA]

@DavidNew-NOAA I explicitly do not want to use the global workflow infrastructure for this, but rather a regression test to validate the results of the code independent of a workflow, such as when we update the hashes weekly. We could possibly use jcb-base.yaml but would still require a preprocessor to define all of the environment variables.

[DavidNew-NOAA]

@CoryMartin-NOAA OK. My preference would be to define all the required environment variables and render jcb-base.yaml.j2, but I'm not married to the idea. Up to you.

[CoryMartin-NOAA]

@DavidNew-NOAA you mean use wxflow and create a jcb-base.yaml from a template?

[DavidNew-NOAA]

@CoryMartin-NOAA Yes, or python Jinja if you consider wxflow to be part of the "workflow"

[CoryMartin-NOAA]

@DavidNew-NOAA thinking about this a bit more now that I'm revisiting it. Are you sure it makes sense to create a template with the variables, define the variables in this script, then generate a YAML rather than just creating the jcb-base.yaml here in the bash script? I don't want to sound like a luddite but it seems like that would require extra steps from doing it this way, but maybe I'm being dense.

[DavidNew-NOAA]

@CoryMartin-NOAA I'm not married to the idea of using the jcb-base.yaml.j2 from GDASApp. It really depends on which infrastructure you're trying to test. Since this is a regression test which tests whether results are changing, likely due to obs or JEDI algorithm changes, than it's probably not necessary. I thought it might make things easier, but if it's over complicating things, let's keep it simple.

[CoryMartin-NOAA]

@DavidNew-NOAA oh I see, you want to use the jcb-base.yaml.j2 that is here: https://github.com/NOAA-EMC/GDASApp/blob/develop/parm/atm/jcb-base.yaml.j2 rather than making another one. I think there are a few things we need to make more modular before we can do that (observations: all_observations for example) but I like that idea longer term