Dev Update: Combining #53, #54, and #55 into one PR#56
Open
taddyb wants to merge 30 commits intodevelopmentfrom
Open
Dev Update: Combining #53, #54, and #55 into one PR#56taddyb wants to merge 30 commits intodevelopmentfrom
taddyb wants to merge 30 commits intodevelopmentfrom
Conversation
* customization of t-route's logging class to comply with logging requirements set in NGWPC-3888 * some log message formatting change * Text file containing repository version number and date * Logger optimizations * Update date in version.txt file * Update release in version.txt file * Removed version, date. Added commit field. No official release has been done yet. * changes to make t-route compatible with v2.2 gpkg * Updated version file for rc-1.0.0 * update version file for release 1.0.0 * Added RTX licensing verbiage * Update version file for release rc-1.1.0 on 2025-02-12 * Update version file for release 1.1.0 on 2025-02-25 * Delete version.txt * PI4 feature integration tests (#8) * added integration tests * removed api data * removed duplicate test * updated LICENSE * PI4 feature unit test updates (#9) * added t-route unit tests * removing outdated tests * upate LICENSE * Pi4 running troute using HF v2.2 (#10) * added files for RnR enhancements to the API, test data, docs * removed unneccessary files, updated LICENSE * JIRA 5916: Port forecast formatting code into T-Route RnR workspace package (#11) * initial commit * patch: updated functions to make code more testable. Added error handling * pr changes * changing name of variable in main to be consistent * feat: pydantic updates * fix: commited pydantic changes too soon --------- Co-authored-by: Quercus <quercus.hamlin@outlook.com> * updated troute-config to move to pydantic v2 (#12) * updated troute-config to move to pydantic v2 * fixed incorrect pyproject * patch: forgot to upload a change in the data assimilation file * fixed incorrect pyproject * missed model rebuild * added comments for why typing tests were removed * JIRA 5920: Integrating the RnR domain logic into `troute-rnr` (#13) * updated troute-config to move to pydantic v2 * fixed incorrect pyproject * patch: forgot to upload a change in the data assimilation file * fixed incorrect pyproject * missed model rebuild * added icefabric hooks to get shortest paths on demand * updated troute-config to move to pydantic v2 (#12) * updated troute-config to move to pydantic v2 * fixed incorrect pyproject * patch: forgot to upload a change in the data assimilation file * fixed incorrect pyproject * missed model rebuild * added comments for why typing tests were removed * JIRA 5917: Replacing the T-Route RnR API with a namespace package (#14) * updated troute-config to move to pydantic v2 * fixed incorrect pyproject * patch: forgot to upload a change in the data assimilation file * fixed incorrect pyproject * missed model rebuild * added icefabric hooks to get shortest paths on demand * added legacy rnr code for formatting input flows * adding t-route support from rnr inputs * moved troute run out of the format file * removed API * updated troute-config to move to pydantic v2 (#12) * updated troute-config to move to pydantic v2 * fixed incorrect pyproject * patch: forgot to upload a change in the data assimilation file * fixed incorrect pyproject * missed model rebuild * added comments for why typing tests were removed * ensuring merge-conflict is resolved properly and code works * JIRA 5918: Validate t-route runs with testing (#15) * updated troute-config to move to pydantic v2 * fixed incorrect pyproject * patch: forgot to upload a change in the data assimilation file * fixed incorrect pyproject * missed model rebuild * added icefabric hooks to get shortest paths on demand * added legacy rnr code for formatting input flows * adding t-route support from rnr inputs * moved troute run out of the format file * added tests for the t-route formatting * got restart test working * updated examples * added new test figure * updated troute-config to move to pydantic v2 (#12) * updated troute-config to move to pydantic v2 * fixed incorrect pyproject * patch: forgot to upload a change in the data assimilation file * fixed incorrect pyproject * missed model rebuild * added comments for why typing tests were removed * JIRA 5917: Replacing the T-Route RnR API with a namespace package (#14) * updated troute-config to move to pydantic v2 * fixed incorrect pyproject * patch: forgot to upload a change in the data assimilation file * fixed incorrect pyproject * missed model rebuild * added icefabric hooks to get shortest paths on demand * added legacy rnr code for formatting input flows * adding t-route support from rnr inputs * moved troute run out of the format file * removed API * updated troute-config to move to pydantic v2 (#12) * updated troute-config to move to pydantic v2 * fixed incorrect pyproject * patch: forgot to upload a change in the data assimilation file * fixed incorrect pyproject * missed model rebuild * added comments for why typing tests were removed * ensuring merge-conflict is resolved properly and code works * added tests for the t-route formatting * got restart test working * removed unneccessary csvs * Jira 5919: Validate T-Route outputs against the max inundation file (#16) * updated troute-config to move to pydantic v2 * fixed incorrect pyproject * patch: forgot to upload a change in the data assimilation file * fixed incorrect pyproject * missed model rebuild * added icefabric hooks to get shortest paths on demand * added legacy rnr code for formatting input flows * adding t-route support from rnr inputs * moved troute run out of the format file * added tests for the t-route formatting * got restart test working * updated examples * adding post_process script for rnr formatting * saving output file * updated troute-config to move to pydantic v2 (#12) * updated troute-config to move to pydantic v2 * fixed incorrect pyproject * patch: forgot to upload a change in the data assimilation file * fixed incorrect pyproject * missed model rebuild * added comments for why typing tests were removed * JIRA 5917: Replacing the T-Route RnR API with a namespace package (#14) * updated troute-config to move to pydantic v2 * fixed incorrect pyproject * patch: forgot to upload a change in the data assimilation file * fixed incorrect pyproject * missed model rebuild * added icefabric hooks to get shortest paths on demand * added legacy rnr code for formatting input flows * adding t-route support from rnr inputs * moved troute run out of the format file * removed API * updated troute-config to move to pydantic v2 (#12) * updated troute-config to move to pydantic v2 * fixed incorrect pyproject * patch: forgot to upload a change in the data assimilation file * fixed incorrect pyproject * missed model rebuild * added comments for why typing tests were removed * ensuring merge-conflict is resolved properly and code works * JIRA 5918: Validate t-route runs with testing (#15) * updated troute-config to move to pydantic v2 * fixed incorrect pyproject * patch: forgot to upload a change in the data assimilation file * fixed incorrect pyproject * missed model rebuild * added icefabric hooks to get shortest paths on demand * added legacy rnr code for formatting input flows * adding t-route support from rnr inputs * moved troute run out of the format file * added tests for the t-route formatting * got restart test working * updated examples * added new test figure * updated troute-config to move to pydantic v2 (#12) * updated troute-config to move to pydantic v2 * fixed incorrect pyproject * patch: forgot to upload a change in the data assimilation file * fixed incorrect pyproject * missed model rebuild * added comments for why typing tests were removed * JIRA 5917: Replacing the T-Route RnR API with a namespace package (#14) * updated troute-config to move to pydantic v2 * fixed incorrect pyproject * patch: forgot to upload a change in the data assimilation file * fixed incorrect pyproject * missed model rebuild * added icefabric hooks to get shortest paths on demand * added legacy rnr code for formatting input flows * adding t-route support from rnr inputs * moved troute run out of the format file * removed API * updated troute-config to move to pydantic v2 (#12) * updated troute-config to move to pydantic v2 * fixed incorrect pyproject * patch: forgot to upload a change in the data assimilation file * fixed incorrect pyproject * missed model rebuild * added comments for why typing tests were removed * ensuring merge-conflict is resolved properly and code works * added tests for the t-route formatting * got restart test working * removed unneccessary csvs * added legacy rnr code for formatting input flows * adding t-route support from rnr inputs * added tests for the t-route formatting * got restart test working * Update log filename handling. Change defalult Python levels ERROR->SEVERE, CRITICAL->FATAL * added uv version of compiler.sh (#17) * added uv version of compiler.sh * removing icefabric from dependencies. Will provide wheels * Patch: remove aws s3 cred references (#18) * added uv version of compiler.sh * removing icefabric from dependencies. Will provide wheels * removed S3/boto3 aws reliance * removed a boto3 reference * Patch 05162025 (#19) * removed redis, fixed copy/paste error * patching post process script and pyyaml imports to ensure code works * PATCH: remove print from settings * Made parsing Gages_ case-insensitive * Patch 06052025: Adding clearer error statements, and solving WFOs which are out of CONUS (#20) * patch: solved error for missing reach id * solved num_messages bug * Update main.py (#21) * JIRA 6401: Formatting the output.csv file in based on OWP suggestions (#22) * updated post_processing to include filtering by time * added a function to format the .nc files after creation to contain RFC metadata * reformatted the post_process file to be what OWP specified * Added a try/catch for t-route (#23) * patch: adding prints for error handling to except value erros * added finally statement to ensure messages are acknowledged * Convert some stdout print msgs to ngen log entries * adding valueError to skip units other than kcfs * adding try-except format to read_rfc_flows * fixed t-route missing hl_link/id/hl_uri * fixed t-route missing hl_link/id/hl_uri * Revert "adding valueError to skip units other than kcfs" This reverts commit 3c36558. * Revert "Revert "adding valueError to skip units other than kcfs"" This reverts commit 97316ce. * Revert "adding valueError to skip units other than kcfs" This reverts commit 3c36558. * Revert " adding try-except format to read_rfc_flows" This reverts commit 8477ad8. * patch: added ValueError handling for incorrect stage * Expose reservoir_assimilation_value to the BMI interface * manually merged hllink fixes * Remove copy/paste error * Fix Downcasting object FutureWarning * Fix HybridParameters and PreprocessingParameters PydanticSerializeUnexpectedValue warnings * Remove unnecessary forward reference double quotes * Remove setuptools-scm from pyproject.toml # Conflicts: # src/troute-config/pyproject.toml * Fix pydantic version in t-route config * Fix hl_reference gpkg layer for lakeout Converted legacy log statements to EWTS log statements --------- Co-authored-by: areg.amirkhanian <areg.amirkhanian@rtx.com> Co-authored-by: Carolyn.Maynard <carolyn.maynard@rtx.com> Co-authored-by: Zhengtao Cui <zhengtao.cui@rtx.com> Co-authored-by: Parallel Works app-run user <yuqiong.liu@mgmt-yuqiongliu-nwmeval-00057.testawsuseast1-1.pw.local> Co-authored-by: Peter Kronenberg <peter.a.kronenberg@rtx.com> Co-authored-by: Quercus <quercus.hamlin@outlook.com> Co-authored-by: Kyle Larkin <kyle.larkin@U-2U1H90H2EWXD5.nextgenwaterprediction.com> Co-authored-by: Kyle Larkin <klarkin@css-inc.com> Co-authored-by: Farshid Rahmani <farshid.rahmani@U-1BVT2XY7DCIBM.nextgenwaterprediction.com> Co-authored-by: yuqiong.liu <yuqiong.liu@ertcorp.com> Co-authored-by: Farshid Rahmani <farshid368@gmail.com>
…servoir flows within the persistence model (#25) * feat: added USBR support for T-route and reservoir persistence modeling * feat: added test subcase for USBR reservoir routing * fix: saved jupyter notebook outputs * fix: renamed test to persistence * fix: patching whitespace
* customization of t-route's logging class to comply with logging requirements set in NGWPC-3888 * some log message formatting change * Text file containing repository version number and date * Logger optimizations * Update date in version.txt file * Update release in version.txt file * Removed version, date. Added commit field. No official release has been done yet. * changes to make t-route compatible with v2.2 gpkg * Updated version file for rc-1.0.0 * update version file for release 1.0.0 * Added RTX licensing verbiage * Update version file for release rc-1.1.0 on 2025-02-12 * Update version file for release 1.1.0 on 2025-02-25 * Delete version.txt * PI4 feature integration tests (#8) * added integration tests * removed api data * removed duplicate test * updated LICENSE * PI4 feature unit test updates (#9) * added t-route unit tests * removing outdated tests * upate LICENSE * Pi4 running troute using HF v2.2 (#10) * added files for RnR enhancements to the API, test data, docs * removed unneccessary files, updated LICENSE * JIRA 5916: Port forecast formatting code into T-Route RnR workspace package (#11) * initial commit * patch: updated functions to make code more testable. Added error handling * pr changes * changing name of variable in main to be consistent * feat: pydantic updates * fix: commited pydantic changes too soon --------- Co-authored-by: Quercus <quercus.hamlin@outlook.com> * updated troute-config to move to pydantic v2 (#12) * updated troute-config to move to pydantic v2 * fixed incorrect pyproject * patch: forgot to upload a change in the data assimilation file * fixed incorrect pyproject * missed model rebuild * added comments for why typing tests were removed * JIRA 5920: Integrating the RnR domain logic into `troute-rnr` (#13) * updated troute-config to move to pydantic v2 * fixed incorrect pyproject * patch: forgot to upload a change in the data assimilation file * fixed incorrect pyproject * missed model rebuild * added icefabric hooks to get shortest paths on demand * updated troute-config to move to pydantic v2 (#12) * updated troute-config to move to pydantic v2 * fixed incorrect pyproject * patch: forgot to upload a change in the data assimilation file * fixed incorrect pyproject * missed model rebuild * added comments for why typing tests were removed * JIRA 5917: Replacing the T-Route RnR API with a namespace package (#14) * updated troute-config to move to pydantic v2 * fixed incorrect pyproject * patch: forgot to upload a change in the data assimilation file * fixed incorrect pyproject * missed model rebuild * added icefabric hooks to get shortest paths on demand * added legacy rnr code for formatting input flows * adding t-route support from rnr inputs * moved troute run out of the format file * removed API * updated troute-config to move to pydantic v2 (#12) * updated troute-config to move to pydantic v2 * fixed incorrect pyproject * patch: forgot to upload a change in the data assimilation file * fixed incorrect pyproject * missed model rebuild * added comments for why typing tests were removed * ensuring merge-conflict is resolved properly and code works * JIRA 5918: Validate t-route runs with testing (#15) * updated troute-config to move to pydantic v2 * fixed incorrect pyproject * patch: forgot to upload a change in the data assimilation file * fixed incorrect pyproject * missed model rebuild * added icefabric hooks to get shortest paths on demand * added legacy rnr code for formatting input flows * adding t-route support from rnr inputs * moved troute run out of the format file * added tests for the t-route formatting * got restart test working * updated examples * added new test figure * updated troute-config to move to pydantic v2 (#12) * updated troute-config to move to pydantic v2 * fixed incorrect pyproject * patch: forgot to upload a change in the data assimilation file * fixed incorrect pyproject * missed model rebuild * added comments for why typing tests were removed * JIRA 5917: Replacing the T-Route RnR API with a namespace package (#14) * updated troute-config to move to pydantic v2 * fixed incorrect pyproject * patch: forgot to upload a change in the data assimilation file * fixed incorrect pyproject * missed model rebuild * added icefabric hooks to get shortest paths on demand * added legacy rnr code for formatting input flows * adding t-route support from rnr inputs * moved troute run out of the format file * removed API * updated troute-config to move to pydantic v2 (#12) * updated troute-config to move to pydantic v2 * fixed incorrect pyproject * patch: forgot to upload a change in the data assimilation file * fixed incorrect pyproject * missed model rebuild * added comments for why typing tests were removed * ensuring merge-conflict is resolved properly and code works * added tests for the t-route formatting * got restart test working * removed unneccessary csvs * Jira 5919: Validate T-Route outputs against the max inundation file (#16) * updated troute-config to move to pydantic v2 * fixed incorrect pyproject * patch: forgot to upload a change in the data assimilation file * fixed incorrect pyproject * missed model rebuild * added icefabric hooks to get shortest paths on demand * added legacy rnr code for formatting input flows * adding t-route support from rnr inputs * moved troute run out of the format file * added tests for the t-route formatting * got restart test working * updated examples * adding post_process script for rnr formatting * saving output file * updated troute-config to move to pydantic v2 (#12) * updated troute-config to move to pydantic v2 * fixed incorrect pyproject * patch: forgot to upload a change in the data assimilation file * fixed incorrect pyproject * missed model rebuild * added comments for why typing tests were removed * JIRA 5917: Replacing the T-Route RnR API with a namespace package (#14) * updated troute-config to move to pydantic v2 * fixed incorrect pyproject * patch: forgot to upload a change in the data assimilation file * fixed incorrect pyproject * missed model rebuild * added icefabric hooks to get shortest paths on demand * added legacy rnr code for formatting input flows * adding t-route support from rnr inputs * moved troute run out of the format file * removed API * updated troute-config to move to pydantic v2 (#12) * updated troute-config to move to pydantic v2 * fixed incorrect pyproject * patch: forgot to upload a change in the data assimilation file * fixed incorrect pyproject * missed model rebuild * added comments for why typing tests were removed * ensuring merge-conflict is resolved properly and code works * JIRA 5918: Validate t-route runs with testing (#15) * updated troute-config to move to pydantic v2 * fixed incorrect pyproject * patch: forgot to upload a change in the data assimilation file * fixed incorrect pyproject * missed model rebuild * added icefabric hooks to get shortest paths on demand * added legacy rnr code for formatting input flows * adding t-route support from rnr inputs * moved troute run out of the format file * added tests for the t-route formatting * got restart test working * updated examples * added new test figure * updated troute-config to move to pydantic v2 (#12) * updated troute-config to move to pydantic v2 * fixed incorrect pyproject * patch: forgot to upload a change in the data assimilation file * fixed incorrect pyproject * missed model rebuild * added comments for why typing tests were removed * JIRA 5917: Replacing the T-Route RnR API with a namespace package (#14) * updated troute-config to move to pydantic v2 * fixed incorrect pyproject * patch: forgot to upload a change in the data assimilation file * fixed incorrect pyproject * missed model rebuild * added icefabric hooks to get shortest paths on demand * added legacy rnr code for formatting input flows * adding t-route support from rnr inputs * moved troute run out of the format file * removed API * updated troute-config to move to pydantic v2 (#12) * updated troute-config to move to pydantic v2 * fixed incorrect pyproject * patch: forgot to upload a change in the data assimilation file * fixed incorrect pyproject * missed model rebuild * added comments for why typing tests were removed * ensuring merge-conflict is resolved properly and code works * added tests for the t-route formatting * got restart test working * removed unneccessary csvs * added legacy rnr code for formatting input flows * adding t-route support from rnr inputs * added tests for the t-route formatting * got restart test working * Update log filename handling. Change defalult Python levels ERROR->SEVERE, CRITICAL->FATAL * added uv version of compiler.sh (#17) * added uv version of compiler.sh * removing icefabric from dependencies. Will provide wheels * Patch: remove aws s3 cred references (#18) * added uv version of compiler.sh * removing icefabric from dependencies. Will provide wheels * removed S3/boto3 aws reliance * removed a boto3 reference * Patch 05162025 (#19) * removed redis, fixed copy/paste error * patching post process script and pyyaml imports to ensure code works * PATCH: remove print from settings * Made parsing Gages_ case-insensitive * Patch 06052025: Adding clearer error statements, and solving WFOs which are out of CONUS (#20) * patch: solved error for missing reach id * solved num_messages bug * Update main.py (#21) * JIRA 6401: Formatting the output.csv file in based on OWP suggestions (#22) * updated post_processing to include filtering by time * added a function to format the .nc files after creation to contain RFC metadata * reformatted the post_process file to be what OWP specified * Added a try/catch for t-route (#23) * patch: adding prints for error handling to except value erros * added finally statement to ensure messages are acknowledged * Convert some stdout print msgs to ngen log entries * adding valueError to skip units other than kcfs * adding try-except format to read_rfc_flows * fixed t-route missing hl_link/id/hl_uri * fixed t-route missing hl_link/id/hl_uri * Revert "adding valueError to skip units other than kcfs" This reverts commit 3c36558. * Revert "Revert "adding valueError to skip units other than kcfs"" This reverts commit 97316ce. * Revert "adding valueError to skip units other than kcfs" This reverts commit 3c36558. * Revert " adding try-except format to read_rfc_flows" This reverts commit 8477ad8. * patch: added ValueError handling for incorrect stage * Expose reservoir_assimilation_value to the BMI interface * manually merged hllink fixes * Remove copy/paste error * Fix Downcasting object FutureWarning * Fix HybridParameters and PreprocessingParameters PydanticSerializeUnexpectedValue warnings * Remove unnecessary forward reference double quotes * Remove setuptools-scm from pyproject.toml # Conflicts: # src/troute-config/pyproject.toml * Fix pydantic version in t-route config * Fix hl_reference gpkg layer for lakeout Converted legacy log statements to EWTS log statements --------- Co-authored-by: areg.amirkhanian <areg.amirkhanian@rtx.com> Co-authored-by: Carolyn.Maynard <carolyn.maynard@rtx.com> Co-authored-by: Zhengtao Cui <zhengtao.cui@rtx.com> Co-authored-by: Parallel Works app-run user <yuqiong.liu@mgmt-yuqiongliu-nwmeval-00057.testawsuseast1-1.pw.local> Co-authored-by: Peter Kronenberg <peter.a.kronenberg@rtx.com> Co-authored-by: Quercus <quercus.hamlin@outlook.com> Co-authored-by: Kyle Larkin <kyle.larkin@U-2U1H90H2EWXD5.nextgenwaterprediction.com> Co-authored-by: Kyle Larkin <klarkin@css-inc.com> Co-authored-by: Farshid Rahmani <farshid.rahmani@U-1BVT2XY7DCIBM.nextgenwaterprediction.com> Co-authored-by: yuqiong.liu <yuqiong.liu@ertcorp.com> Co-authored-by: Farshid Rahmani <farshid368@gmail.com>
* initial commit: implemented forcing reads for ET * feat: added remaining ELOSS and SSOUT code * fix: addressing PR comments
* initial commit: addressed comments from PI-6 Deliverable review from github * fix: removed compiler_uv.sh as UV has been integrated into the base compiler * patch: updating rnr with icefabric segments * fix: pyproject was not importing correctly with icefabric dependencies * patch: fixed legacy icefabric imports
* fix: removed all icefabric dependencies and added rnr subsetting code * patch: added parquet/ sub-folder * patch: added parquet subfolder to post-process script
* feat: added iac entrypoint to troute-rnr * feat: converted iac entrypoint to a arg * docs: updated readme
* patch: added s3fs import only if using IaC * patch: fixed parquet typo and post-processing logic * patch: ensured hydrolocations is not empty when being passed
* patch: updated boto3 version * patch: changed boto3 version
* fix: removed xarray s3 writes * fix: removed trailing / * fix: writing nc to disk before uploading
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Given there is too much code in #53 , #54, and #55 that conflicts, I did all of the in-order merging into this branch to merge back to dev.