Feature/parse obs schema variable dimension field#9
Draft
amstokely wants to merge 85 commits intoibanos90:mainfrom
Draft
Feature/parse obs schema variable dimension field#9amstokely wants to merge 85 commits intoibanos90:mainfrom
amstokely wants to merge 85 commits intoibanos90:mainfrom
Conversation
…me comments for documentation
**Description** This PR fixes some bugs found to convert raw AHI observations from the AWS cloud in Himawari Standard Data (HSD) format to the IODA-v2 needed in `mpas-jedi`. It was added the capability to read files with segment number 01, which corresponds to the no segment division of HSD data. The reading of the variables was modified to read each one explicitly since otherwise it was not working properly (not sure why). Specifically the way the variable `idata`, which is the count (total number of pixels that results from multiplying the number of columns by the number of lines) needed for the radiance calculation is also modified. This modification results in the correct calculation of the radiance and then the brightness temperature. In addition, the variable for the `dateTime` in `epochtime` type was allocated since that was missing. The observation error was also added following the values used in GSI (https://www.nco.ncep.noaa.gov/pmb/codes/nwprod/gfs.v16.2.2/fix/fix_gsi/global_satinfo.txt) for clear-sky and a bug in the function that sets the errors was fixed as well. **Closes** Issue #2
This PR adds the superobbing capability for AHI observations. Giving the half width of the superobbing grid as an entry parameter, the code computes the average brightness temperature in the box defined using that parameter. This code is adapted from https://github.com/wrf-model/WRF/blob/develop/var/da/da_radiance/da_read_obs_netcdf4ahi_jaxa.inc and follows the same implementation as for ABI observations from GOES-16. The debugging build flags for the goes_abi code are removed. The following figures show the brightness temperature results for AHI observations after using this converter for 2018-04-15T00 channel 8 (a) without superobbing and (b) with superobbing on a grid of 59x59 pixels: **a)**  **b)**  **Issue closed** Closes #1
Revised define_mod.f90 with adding obs errors for CrIS-FSR. For data after 2021: Observation errors for the CrIS-FSR with 431 channels have been added. For data before 2021: If you are processing CrIS data with 399 channels, you will need to manually add the observation errors. The obs error assigned here are from the two obs file in UFO (It's easy to copy and paste): ufo-data/testinput_tier_1/cris-fsr_n20_obs_2020121500_m.nc4 ufo-data/testinput_tier_1/cris-fsr_npp_obs_2018041500_m.nc4 The cris-fsr_npp 's errors in this PR are the same as the following file https://www.nco.ncep.noaa.gov/pmb/codes/nwprod/gfs.v16.3.16/fix/fix_gsi/global_satinfo.txt while cris-fsr_n20 's errors are different from global_satinfo.txt.
This PR modifies the documentation on converting the HSD file using obs2ioda. Default value for pixel to skip (option -s) has been changed from 3 to 1 which means no pixel will be skipped. Modified Files: M obs2ioda-v1/README.md M obs2ioda-v2/README.md
TYPE: enhancement KEYWORDS: cmake, build system, compiler compatibility DESCRIPTION OF CHANGES: The obs2ioda-v2 library lacked support for CMake-based builds, which complicated the build process across different platforms and made compiler compatibility (GNU and Intel) more difficult to manage. Solution: 1. Added support for building obs2ioda-v2 using CMake, leveraging the modern build system to streamline cross-platform integration. 2. Updated the obs2ioda-v2 README with instructions on how to build the library using CMake and provided additional guidance for building the NCEP BUFR library. 3. Fixed a boolean equality check in hsd.f90 to ensure obs2ioda-v2 compiles successfully on both GNU and Intel compilers. These changes improve the ease of configuration and build process, while maintaining compatibility with multiple compilers.
…ms bounds checking and saves additional debug info when obs2ioda is compiled in Debug mode.
Prevented out-of-bounds memory access and inaccurate calculations caused by the uninitialized iloc variable, used when parsing AHI files without superobs.
Fix Segmentation Fault and Memory Access Issues in AHI File Parsing by Initializing iloc
…ging options (#21) This PR introduces standardized Fortran compiler flags tailored for the GNU and Intel compilers, enhancing both build consistency and debugging capability across all build types. The updates are as follows: GNU Compiler Flags: General: -ffree-line-length-none (removes line length limits), and -mcmodel=medium (supports larger memory models for larger datasets). Debug: Aggressive debugging flags including -fbacktrace (provides error backtrace), -ffpe-trap=invalid,zero,overflow (traps floating-point exceptions), and -fcheck=all (enables all runtime checks). Intel Compiler Flags: General: -mcmodel=medium (supports larger memory models for larger datasets.) Debug: Includes -check uninit (checks uninitialized variables), -ftrapuv (traps uninitialized variable usage), -g (produces debug information), -traceback (provides symbolic traceback on errors), and -fpe0 (halts on floating-point exceptions). These flags standardize the compilation environment, making it easier for developers to catch issues, particularly in Debug builds where aggressive error-trapping is employed.
…interoperability between Fortran and C. (#30) An example of how this string interface could be used in the context of writing a NetCDF string array variable is shown as follows. Given the signature of a C/C++ function that writes a NetCDF string array variable: int netcdfPutVarString( int netcdfID, const char *groupName, const char *varName, const char **data ); , which maps to the following fortran iso c binding interface: function c_netcdfPutVarString(& netcdfID, groupName, varName, data) & bind(C, name = "netcdfPutVarString") import :: c_int import :: c_ptr integer(c_int), value, intent(in) :: netcdfID type(c_ptr), value, intent(in) :: groupName type(c_ptr), value, intent(in) :: varName type(c_ptr), value, intent(in) :: data integer(c_int) :: c_netcdfPutVarString end function c_netcdfPutVarString , f_c_string_1D_t and f_c_string_t would be used as follows: function netcdfPutVarString(& netcdfID, groupName, varName, data) result(status) integer, value, intent(in) :: netcdfID character(len=:), allocatable, intent(inout) :: groupName character(len=:), allocatable, intent(inout) :: varName character(len=:), allocatable, dimension(:), intent(inout) type(f_c_string_t) :: f_c_string_groupName type(f_c_string_t) :: f_c_string_varName type(f_c_string_t_1D) :: f_c_string_data integer :: status status = C_netcdfPutVarString(& netcdfID, & f_c_string_groupName(groupName)%to_c(), & f_c_string_varName(varName)%to_c(), & f_c_string_data(data)%to_c()) end function c_netcdfPutVarString
… simplify and standardize the configuration of obs2ioda C++ library targets. (#32) Function Overview The obs2ioda_cxx_library function takes a single argument, target, representing the C++ target to configure. It performs the following key actions: Sets the INSTALL_RPATH property to ensure that shared libraries can be found relative to the target's installation directory. Retrieves public link libraries by dynamically constructing the variable name ${target}_PUBLIC_LINK_LIBRARIES. Links the retrieved libraries to the target using target_link_libraries. This setup ensures proper linkage and runtime configuration for relocatable installations. Benefits Improved Modularity: Centralizes and simplifies the configuration of C++ libraries in obs2ioda. Consistency: Ensures all C++ libraries follow the same setup process for linking and runtime path configuration. Ease of Use: Reduces repetitive CMake code by encapsulating common configuration steps in a reusable function.
…obs2ioda_fortran_library and obs2ioda_fortran_executable. (#31) Previously, the function configured both Fortran libraries and their corresponding executables, limiting flexibility. This update introduces separate functions for libraries and executables, removing their dependency and enabling standalone library creation. These changes improve modularity, clarify function use, and enhance maintainability of the CMake build system.
…33) * Implemented a function, netcdErrorMessage, to handle NetCDF C++ exceptions. netcdfErrorMessage prints the netcdf exception's message to the terminal and returns the corresponding NetCDF error code. * Reworked netcdfErrorMessage function to provide more info. * Removed duplicate obs2ioda_cxx_library def * Removed duplicate definition of obs2ioda_cxx_library. * Updated formatting so error messages are only two lines.
* Implemented interface for calling C++ NetCDF's file operations api from fortran. * Updated createFile to allow the user to set the netcdf filemode. * Fixed file mode cast error and fileMode 'value' error in the fortran interface. * Changed add/removeFile methods from int to void. * Updated add/removeFile docs. * Updated docs.
… API (#35) This PR introduces support for creating NetCDF groups and adding them to existing NetCDF files from Fortran using the NetCDF C++ API. The implementation provides a user-friendly Fortran interface that abstracts the complexities of memory management and C/Fortran data type mappings. Key Features: NetCDF Group Creation: Added functionality to create new groups within NetCDF files. Supports adding groups to specific parent groups or directly to the root group. User Friendly Fortran Integration: Memory management is handled entirely within the Fortran interface, reducing the need for user intervention. Automatic mapping between Fortran strings and null-terminated C strings to ensure compatibility. Motivation: This enhancement bridges the gap between Fortran applications and advanced NetCDF capabilities provided by the C++ API. It enables users to work with hierarchical group structures in NetCDF files directly from Fortran without dealing with low-level details. Next Steps: Extend the interface to support additional operations: Adding and retrieving attributes, variables, and data. Enhanced error reporting and diagnostics.
Fix satellite ID bug in gnssro_module
Enable hourly IODA file output for GNSSRO obs
Add support for single-precision floating point array attributes.
Summary This PR refactors the ncio_mod module by replacing direct NetCDF Fortran calls with a NetCDF C++ wrapper. Key Changes NetCDF I/O Update: Replaced all NetCDF Fortran I/O calls with equivalent calls from the NetCDF C++ wrapper (netcdf_cxx).
* Replace netcdf calls with new interface. * Remove commented out calls to old netcdf interface * Remove legacy group and variable identifiers * Update call signature of netcdfAddDim to include dimension id. * Add integer array attributes to IODA file. * Clean up after rebase off updated main * Add floating-point array attributes to gnssro ioda file * Move fill value definition from separate routine to variable definition routine.
* Rewrote FindNetCDF.cmake so it works on MacOs and Linux. * Fixed NetCDF_Found logic
* Created qa branch. * Create global vars for all dims. * Create global var for each dim. * Reverted to original logic for creating nchans dim var. * Added ws around if statement ( and ). * Fixed dimension variable name error in write_obs. * Simplified netcdfPutAtt call in write_obs. * Updated logic for string array variables that formerly relied on the variable being an nf90_char. * Fixed coordinate variable dimension name argument bug. Rebased on `main` to undo changes in `hsd.f90`, `prepbufr_mod.f90`, `radiance_mod.f90`, and `satwnd_mod.f90`, which came from the `feature/char_to_string_netcdf_vars` branch.
…option when building on Linux, as this flag is not defined by Apple Clang. (#59) Co-authored-by: Andy Stokely <amstokely>
* Move scan_position type conversion to ncio_mod. parent 2ab69a3 This PR updates NetCDF variable handling by converting character array variables to string array variables, ensuring compliance with the IODA v2 format. Changed datetime variable type from char to string. * Fixed allocation error. * Fixed deallocation bug. * Changed datetime back to being a nf90_char vs nf90_string. * Moved all logic related to converting nf90_char to nf90_string variables to ncio_mod. This limits all code changes to within ncio_mod.
…ake. Also configured CMake to build googletest. (#61)
* Configured CMakeLists.txt in obs2ioda/src/cxx to build yaml-cpp. * Change yaml-cpp tag from master to 0.8.0 and added include(FetchContent) call to root CMakeLists.
* This PR introduces the `IodaObsSchema` class, which parses the `ObsSpace.yaml` IODA format validation file and uses its contents to validate `obs2ioda` NetCDF output files against the latest IODA format naming and formatting conventions. Key features of `IodaObsSchema`: - Ensures `obs2ioda` IO remains compliant with evolving IODA naming conventions. - Allows `obs2ioda` to continue using its original IO logic while benefiting from schema-based validation. - Helps maintain consistency with the latest IODA standards without requiring frequent manual updates to `obs2ioda` output logic. These changes were tested by generating observation files that successfully passed the **3DEnVar** and **HofX** tutorial calculations. Temp removed tests. * Added tests for IodaObsSchema class. * Added test subdirectory to root CMakeLists.txt * Updated C++ standard. This required for building yaml-cpp on a Mac. * Removed duplicate yaml-cpp fetch content call. * Removed duplicate yaml-cpp fetch content call. * Set C++ standard to 17. This is required for building yaml-cpp on a Mac. * Moved def of IODA_SCHEMA_YAML to ioda_obs_schema.h. * Removed duplicate FetchContent(googletest) and include(FetchContent) calls. * Created macro, set in cmake, that stores the obs2ioda root directory path. This is used to define the ObsSpace.yaml path. * Moved constructor def for IodsObsVariable out of header. * Created a config header for setting the ioda schema yaml path at compile time. This approach avoids the use of macros and will produce a compile vs. runtime error if the ioda schema file does not exist. --------- Co-authored-by: Andy Stokely <amstokely>
* Deleted GMake Makefile. * Updated obs2ioda executable name to obs2ioda_v3. * Updated a bunch of names. * Updated README * Updated README links. * Update CMakeLists.txt * Update CMakeLists.txt * Update README.md * Update README.md * Update README.md * Update README.md
… env file that are used for setting validation test related file paths at compile time. This allows the validation tests to be run from the build directory and also allow you to define the build directory outside of the obs2ioda source directory.
…of IodaObsSchema.
…file schema pointer when it is no longer needed.
…the ObsSpace.yaml.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.