Add csv post process for time series variables #440

genevievestarke · 2026-01-08T16:33:45Z

Add csv post process for time series variables

This pull requests adds a function to save specified time series variables to a csv file for post processing. The function is meant to only save time series variables from a single H2I run. It loads the data from an sql file, so the outputs of the H2I run need to be saved with a recorder. You can save all the time series variables from a run by not specifying specific variables, or you can enter a dictionary or list of the time series variables you would like to save. You can also provide units and/or alternative names for the variables. The output is a dictionary of the time series variables, and a csv file if the save_to_file input is set to True.

Additionally, there is a minor bug fix to return the recorder path.

Section 1: Type of Contribution

Section 3: General PR Checklist

PR description thoroughly describes the new feature, bug fix, etc.
Added tests for new functionality or bug fixes
Tests pass (If not, and this is expected, please elaborate in the Section 6: Test Results)
Documentation
- Docstrings are up-to-date
- Related docs/ files are up-to-date, or added when necessary
- Documentation has been rebuilt successfully
- Examples have been updated (if applicable)
CHANGELOG.md has been updated to describe the changes made in this PR

Section 3: Related Issues

Section 4: Impacted Areas of the Software

Section 4.1: New Files

h2integrate\postprocess\sql_timeseries_to_csv.py: tests for all possible different use cases of the function save_case_timeseries_as_csv
- check_get_units_for_var: Check the units for a variable within a case and returns the variable value in the specified units and the specified units.
- save_case_timeseries_as_csv: Summarize timeseries data from a case within an sql recorder file to a DataFrame
  and save to csv file if save_to_file is True.
h2integrate\postprocess\test\test_sql_timeseries_to_csv.py

Section 4.2: Modified Files

h2integrate/core/pose_optimization.py
- set_recorders(): minor bugfix to return recorder path if recorder_attachment=="model"
examples/02_texas_ammonia/driver_config.yaml: updated to user recorder
examples/02_texas_ammonia/run_texas_ammonia_plant.py: updated to showcase saving timeseries functionality

Section 5: Additional Supporting Information

Section 6: Test Results, if applicable

elenya-grant

This seems like a really great tool to save timeseries profiles! Thank you! I have a few suggestions and would be happy to chat more about other possible considerations. For example,

The input to the function could be the .sql filepath and the case index then the filename could be output as the same as the sql filename but includes the case number:

output_fpath = sql_fpath.parent / f"{sql_fpath.stem}_Case{case_index}.csv"

Would there be a case where someone would want to save all the timeseries data? We could make this function also able to save all the timeseries variables if a specified name list isn't input.

elenya-grant · 2026-01-08T16:44:16Z

h2integrate/postprocess/timeseries_to_csv.py

+    # get the cases as a list
+    cases = list(cr.get_cases())
+
+    variable_list = [


I think that "variable_list" and "units_list" may be easier to read if it was a dictionary. Like:

variable_to_units_dict = {'battery.SOC': 'percent', 'battery.electricity_in': 'MW'}

elenya-grant · 2026-01-08T16:53:18Z

h2integrate/tools/save_to_csv_partial.py

+            name_list = var.split(".")
+            name_list.append(units_list[loc])
+            save_key = " ".join(name_list)
+        data[save_key] = case.get_val(var, units=units_list[loc])


If you wanted to make this flexible enough to handle a case where a user doesn't specify units, you can get units for a variable using: case._get_units(var)

Also - if you wanted to be consistent with the column naming in convert_sql_to_csv_summary() (h2integrate/postprocess/sql_to_csv.py) then I think the save_key would be formatted as f"{var} ({data_units})" where var is the variable naming that you have in variable_list.

I also think it'd be good to have a check that each variable is an array or else you may get errors from pandas if trying to make a data frame of 8760s but one of the variables is a scalar.

elenya-grant · 2026-01-08T16:53:42Z

h2integrate/tools/save_to_csv_partial.py

+import pandas as pd
+
+def save_as_csv(variable_list, units_list, case, filename, alternative_name_list=None):
+    data = pd.DataFrame()


I know this is a draft but just a reminder to include docstrings!

…ess_csv' into feature/post_process_csv

…ure/post_process_csv

…ame logic

johnjasa

Fun PR! I'm really happy to see you, @genevievestarke, contributing to the code in nice, digestible chunks as you work through a bigger behemoth in #407.

I have some comments and questions that I've marked. The main quasi-blocking one is on if the number of outputs saved is intentional. I left more info in a bigger comment with an example output showing what I mean.

I made some small edits directly to the branch, too.

My other suggestion is to add at least a mention of this functionality to the docs (postprocessing_results.md). A bigger inclusion could show using it within an .ipynb file but that isn't strictly necessary.

Lastly, let me acknowledge that this is a more in-depth review than is purely needed here, but really because I want to give good feedback and make sure this is optimally useful for users. I love the addition and I think it helps answer the question "so I just ran H2I... now what?" so that's very helpful! Thank you!

examples/02_texas_ammonia/run_texas_ammonia_plant.py

h2integrate/postprocess/sql_timeseries_to_csv.py

elenya-grant

This looks great! Thank you for adding this in! I gave one non-blocking comment but this looks good to go!

elenya-grant · 2026-01-20T21:33:33Z

h2integrate/postprocess/test/test_sql_timeseries_to_csv.py

+        assert all(c_name in res.columns.to_list() for c_name in expected_colnames)
+
+
+def test_alternative_column_names(subtests, run_example_02_sql_fpath):


Love this new test! Thank you!

elenya-grant · 2026-01-20T21:35:43Z

docs/user_guide/postprocessing_results.md

+
+### Saving outputs
+
+The time series outputs can be saved to a csv output using the `save_case_timeseries_as_csv` function. If no variables are specified, then the function saves all time series variables in the simulation. Otherwise, the specified variables are saves.


(Non-blocking): could be good to document the different inputs (like the cases done in the tests for this function) here. I'm thinking about specifying units and/or giving it alternative names.

johnjasa

Looks great, thanks for this meaningful quality-of-life improvement to H2I, @genevievestarke! I really like the functionality this adds and I appreciate the effort you put into iterating on it.

I pushed up some doc changes based on Elenya's last non-block suggestion. Great to come in and I'll merge it after tests pass!

genevievestarke added 4 commits January 8, 2026 11:31

Add csv post process for time series variables

dde7f2b

Rename and move to postprocesss folder

7e9c7d5

Clean up function file

329160a

Fis ruff formatting

b9c6d73

elenya-grant requested changes Jan 8, 2026

View reviewed changes

elenya-grant and others added 12 commits January 8, 2026 14:05

added alternative approach to saving timeseries data

b0f65bf

added docstrings and comments to new methods

5e307b5

Add column renaming ability

cc36149

Merge remote-tracking branch 'refs/remotes/personal/feature/post_proc…

47408a1

…ess_csv' into feature/post_process_csv

Add alternative varibale name capability

2759697

bugfix in set_recorders

e8b4538

added timeseries saving to example 2 and removed post_process

86bef92

removed post_process

d1b300a

Fix ruff issue, delete old code

286249f

Sort import statements

68c5bd5

Unsort import statements

138daf3

Add line

dcbd69c

genevievestarke requested review from elenya-grant and johnjasa January 8, 2026 22:47

genevievestarke added the ready for review This PR is ready for input from folks label Jan 8, 2026

elenya-grant and others added 8 commits January 8, 2026 16:18

added basic tests for saving timeseries results

ab79788

Merge remote-tracking branch 'gen/feature/post_process_csv' into feat…

3540f77

…ure/post_process_csv

Add test for alternative column names and adjust alternative column n…

bf8119c

…ame logic

fix formatting

0690fc9

update test name

ac3a91e

Add unit and alternative name test

78a9b22

updated CHANGELOG

3276716

renamed new files and updated import statements

cb26f81

genevievestarke added enhancement New feature or request Tools/Utilities Tools and utilities that are used in H2I or for pre-processing or post-processing labels Jan 9, 2026

genevievestarke added the bug Something isn't working label Jan 9, 2026

genevievestarke marked this pull request as ready for review January 9, 2026 19:32

johnjasa requested changes Jan 9, 2026

View reviewed changes

johnjasa and others added 4 commits January 9, 2026 13:55

Minor comment updates

6726c44

Merge branch 'develop' into feature/post_process_csv

4603508

minor updates based on feedback

3bb1066

Merge branch 'develop' into feature/post_process_csv

c1c9a64

johnjasa mentioned this pull request Jan 13, 2026

Expand H2Integrate's postprocessing utilities #111

Closed

johnjasa and others added 4 commits January 13, 2026 15:00

Merge branch 'develop' into feature/post_process_csv

b5f0f92

Merge branch 'develop' into feature/post_process_csv

2f3c3e7

Add to docs

9607521

Update Ex 2 and docs

38084aa

genevievestarke requested a review from johnjasa January 16, 2026 21:46

elenya-grant added 2 commits January 20, 2026 10:52

minor updates to doc page for formatting

32d5106

updated test since merged in develop

4a5dc35

elenya-grant approved these changes Jan 20, 2026

View reviewed changes

Added doc content to address review feedback

5ce318c

johnjasa approved these changes Jan 21, 2026

View reviewed changes

johnjasa enabled auto-merge (squash) January 21, 2026 05:37

johnjasa and others added 2 commits January 21, 2026 09:03

Merge branch 'develop' into feature/post_process_csv

255e9b5

updated tests for changes to example

6fb4e6c

johnjasa merged commit f3f18ec into NatLabRockies:develop Jan 21, 2026
5 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add csv post process for time series variables #440

Add csv post process for time series variables #440

Uh oh!

genevievestarke commented Jan 8, 2026 •

edited

Loading

Uh oh!

elenya-grant left a comment

Uh oh!

elenya-grant Jan 8, 2026

Uh oh!

elenya-grant Jan 8, 2026

Uh oh!

elenya-grant Jan 8, 2026

Uh oh!

johnjasa left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elenya-grant left a comment

Uh oh!

elenya-grant Jan 20, 2026

Uh oh!

elenya-grant Jan 20, 2026

Uh oh!

johnjasa left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		assert all(c_name in res.columns.to_list() for c_name in expected_colnames)


		def test_alternative_column_names(subtests, run_example_02_sql_fpath):


		### Saving outputs

		The time series outputs can be saved to a csv output using the `save_case_timeseries_as_csv` function. If no variables are specified, then the function saves all time series variables in the simulation. Otherwise, the specified variables are saves.

Add csv post process for time series variables #440

Add csv post process for time series variables #440

Uh oh!

Conversation

genevievestarke commented Jan 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Add csv post process for time series variables

Section 1: Type of Contribution

Section 3: General PR Checklist

Section 3: Related Issues

Section 4: Impacted Areas of the Software

Section 4.1: New Files

Section 4.2: Modified Files

Section 5: Additional Supporting Information

Section 6: Test Results, if applicable

Uh oh!

elenya-grant left a comment

Choose a reason for hiding this comment

Uh oh!

elenya-grant Jan 8, 2026

Choose a reason for hiding this comment

Uh oh!

elenya-grant Jan 8, 2026

Choose a reason for hiding this comment

Uh oh!

elenya-grant Jan 8, 2026

Choose a reason for hiding this comment

Uh oh!

johnjasa left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elenya-grant left a comment

Choose a reason for hiding this comment

Uh oh!

elenya-grant Jan 20, 2026

Choose a reason for hiding this comment

Uh oh!

elenya-grant Jan 20, 2026

Choose a reason for hiding this comment

Uh oh!

johnjasa left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

genevievestarke commented Jan 8, 2026 •

edited

Loading