possible Model cycling(NoDA) issue

[guoqing-noaa]

We recently find the model cycling issue may come back. Haidao, Chunhua, Ruifang and my retros all confirmed a similar behavior: forecasts from the hourly "mpasout" cycling (No DA) are evidently different from the cold start forecasts valid at the same time. We expect they should be very close. This means the model cycling issue may come back.

To confirm this, I further tested the latest ufs-community/MPAS-Model (version v8.3.0-1.13). I removed the packages attributes for relevant variables in the da_state stream, so no more warning messages as reported in #150. Hence I assume the mutable/immutable choice of da_state no longer affects the cycling test results.

Here is the slides of the MATS verification where cycling(NoDA) 1h forecasts from 11/23z are compared with cold start 12h forecasts from 00/12z:
https://docs.google.com/presentation/d/1G09dWTjhYYdOjrjoX6nXs7HQofP5DwgIdLF083DFFDY/edit?slide=id.p#slide=id.p

For convenience, here is the verification plot against the sounding data:

We will do more tests trying to digging this more.
But it will be good to get some inputs from the model experts. Thanks!
@clark-evans @barlage @AndersJensen-NOAA @joeolson42 @hu5970

[clark-evans]

Hi @guoqing-noaa -- thanks for sharing. @barlage has extended his code testing framework to include restart tests. The results in #154 document where we are. The hrrrv5 RAP summer case was reproducing through ~8.3.0-1.0 apart from some diagnostics. Mike's going to check to see if it's 8.3.0-1.1 that led to this breaking sometime between there and 8.3.0-1.8.

Winter cases, or more generally seemingly anything involving sea ice, do not reproduce and have not since at least 8.2.2-1.0. We're concerned that the root is in the RUC LSM, which will take us longer to isolate and debug.

[guoqing-noaa]

@clark-evans Thanks for the information.
Thanks @barlage for digging into the reproducibility issue.
I will also do a NoDA cycling test on 8.3.0-1.0 to see how it goes on the cycling part.

[guoqing-noaa]

I did a test of 8.3.0-1.13 using restart.nc in cycling (slides). The differences are much smaller than using mpasout.nc in cycling but they are still noticeable in 600hPa ~ 300hPa in RH, 700hPa ~ 400hPa in wind.

mpasout cycling - RMS and bias:

resart cycling - RMS and bias:

[clark-evans]

@guoqing-noaa would it be worth replicating these tests with the v8.3.0-1.15 code to see if that addresses the issue? Otherwise, a fuller fix will probably have to wait until #162 is addressed.

[guoqing-noaa]

@guoqing-noaa would it be worth replicating these tests with the v8.3.0-1.15 code to see if that addresses the issue? Otherwise, a fuller fix will probably have to wait until #162 is addressed.

I will give it a try. Thanks, @clark-evans

[clark-evans]

#210 provides a new implementation of MYNN surface layer that is based on the CCPP version, which is known to reproduce. It may be worth pulling that down and testing it if these issues still persist.