Historical stations - need to implement web services

[smalers]

The historical web services need to be implemented for surface water and climate stations.

[smalers]

While implementing TSTool features to use CDSS web services for historical station data, I have the following questions and observations. I will update this issue as I work on the implementation. Given that I did not have budget to implement features when web services first came out, the time may have passed for the State to respond to feedback. However, at a minimum, maybe web service documentation could be enhanced to explain nuances and maybe the State has its own issue list internally that could benefit from some of the following. I expect to have features working in the next couple of days and will make decisions along the way. As usual, I will update the TSTool datastore appendix documentation to reflect the implementation.

Observations and Notes

1. Feedback on MeasType:

   1. surfacewaterstationdatatypes and climatesttiondatatypes services both return some of the same "measType". Should Streamflow be returned for surface water stations and the others climate stations? For now I will implement the code as best I can. Note also that some of the data types such as MaxTemp has spaces at the end, resulting in redundant data types.

   Climate station data types:
   
     "measType": "Evap",
     "measType": "FrostDate",
     "measType": "MaxTemp   ",
     "measType": "MaxTemp",
     "measType": "MeanTemp",
     "measType": "MinTemp   ",
     "measType": "MinTemp",
     "measType": "Precip",
     "measType": "Snow",
     "measType": "SnowDepth",
     "measType": "SnowSWE",
     "measType": "Solar",
     "measType": "Streamflow",
     "measType": "VP",
     "measType": "Wind",
   
   Surface water station data types:
   
     "measType": "MaxTemp",
     "measType": "MeanTemp",
     "measType": "MinTemp",
     "measType": "Solar",
     "measType": "Streamflow",
     "measType": "VP",
     "measType": "Wind",
   

2. Feedback on dateFormat:

   1. It would be helpful if the API documentation were more clear as to defaults. For example, what is the date/time default for dateFormat? Also, for data that are dates, why not make the default format the date. There can be confusion as to whether midnight (which is time zero of a day) corresponds to a computed value over the previous day. Using date format would clear this up.
   2. Perhaps specifying dateFormat should not change modified since that is always a time? In particular if someone is requesting monthly or annual data for an automated data dump, it would be good to check the actual modification time, not the month or year from the modification time.
   3. For month and year data, the date is in parts but if were provided in ISO 8601 form, it would be good to default to precision of the data.
   4. Annual streamflow data are in water year. OK, but that is more constrained than other web services. Maybe have a parameter that would allow returning in calendar year. I am pondering whether TSTool should ask for monthly time series and convert to year interval, or whether the data type should indicate water year.

3. Feedback on measCount:

   1. Would be good in documentation to indicate what this is. For monthly data it appears to be the number of daily values used in computing statistics. For year interval data it appears to be the same, which implies that year interval data are computed from day data.

4. Feedback on value:

   1. For time series data, it would be helpful to indicate in API documentation that only dates with a measurement or calculated value are returned. In other words, value is always a valid number and consuming code does not need to deal with a missing value indicator such as null, NaN (JSON does not actually use), or -999 (which can be valid valid value for some time series).
   2. I am using objects in the code and can handle setting to null for missing so even if I don't understand how missing is handled for everything, hopefully using nulls will work in most cases.

5. Feedback on climate station frost dates:

   1. The frost dates values do not contain. Since the climate station data types don't contain frost dates, I am going to assume that other temperature data are at the same stations to get the agency.
   2. The frost dates records do not contain siteId or any other human-facing identifier. This can be confusing when more than one station's data are read.

6. Feedback on surfacewaterstationdatatypes service:

   1. Does not allow query by abbrev.

7. Feedback on climatestations:

   1. Uses latitude when other services use latdecdeg
   2. Uses longitude when other services use longdecdeg

8. Feedback on surfacewaterstations:

   1. Uses latitude when other services use latdecdeg
   2. Uses longitude when other services use longdecdeg
   3. Why is measUnit included when that is specific to data types?

Design Approach

It is not possible to fully rely on web services to provide a clean list of data types for use in TSTool and therefore some manual handling in the code is necessary. Some design decisions are as follows:

1. It appears that the web services data are similar to the previous HydroBase data and therefore I am trying to use similar conventions for both.
2. Each major service is indicated by a group in the Data Type selector. This makes it relatively straightforward to cross reference TSTool with web services. See the example below for Surface Water Station.
3. Some of the datatypes being used in web services are indicated by a main measType and then an interval value computed as a statistic, for example for monthly streamflow measType=Streamflow and a data value of minQCfs. In the TSTool world, this combines main data type, statistic, and data units. To make this work in TSTool, I am going to use a data type of Streamflow-Min and set the units as appropriate. See the example below.
4. The concept of statistic when applied to interval data can be applied generally. However, I need to decide if this makes sense based on some experiments to see what data is available. Things can be confusing. I can implement the functionality for people to access the data, but my explanation of the data may be incomplete or inaccurate if I don't have all the explanation. For example:
   1. For climate station Evap are minimum and maximum values for a month the minimum and maximum daily values in the month? I think yes.
   2. For MaxTemp, it is really daily maximum temperature, so a monthly time series for minValue would be the monthly minimum of daily maximum temperature, right?
   3. Are the month and year interval values for streamflow based on analysis of "instantaneous" or daily values? For example, is month time series minQCfs the minimum of instantaneous streamflow, or minimum of the average daily flows? This is important for peak flow analysis.
   4. Better documentation explaining the measType and *value handling in web services would be helpful.
5. Units are specified in different ways. Sometimes they are in station object, sometimes in data type object, and sometimes in the time series records. Sometimes, as in monthly streamflow, the units are in the value name like minQCfs. All of this is handled in TSTool but it is not consistent in the services.
6. Telemetry stations have open-ended list of data types and a lot more variability, so they are the fall-through when listed in TSTool.

[smalers]

After implementing the web services, automated tests, and documentation, here are the remaining questions and comments that I have for the State. Other feedback above is still relevant, such as the need for improved documentation for some of the details. The automated tests compare HydroBase database datastore and REST web services datastore results because otherwise it is difficult to know if the REST web service values are as expected.

The following is an example of a Linux command that can be used to check for distinct measType in the data types results:

grep measType /C/Users/sam/Downloads/download-climate-datatypes.json | sort -u

1. Climate station data types:
   1. Climate station data types still include Streamflow - is this a data issue? From Doug: [Fixed] I confirmed.
   2. Surface water station data types now only includes Streamflow, which is good.
   3. Now that the data types have been cleaned up,
      TSTool does queries up front to get a unique list of measType to use in data type choice.
      This seems to be a bit slow for surface water stations.
      It would be useful to have a service for the unique list of measType each for
      climate stations and surface water stations to increase performance. From Doug: [Fixed]. Are you using GET api/v2/climatedata/climatestationsdatatypes and GET api/v2/surfacewater/surfacewaterstationdatatypes? They return a unique list of measType for each station. Or are you asking for 2 new services that just return a unique list of measType for ClimateStation / Surface Water Stations regardless of station? I have been doing a query like Doug indicated and now that the response is fast, I retract my request for a new service.
   4. Would it be possible to set the measUnit on frost dates, such as to day? Currently the units are blank. From Doug: [Done] I verified.
   5. The dataSource has CoAgMet and COAGMET. From Doug: [Fixed] I verified.
      I have not checked to see whether the same station is loaded with both but it would be good to use only one abbreviation.
   6. Climate station data types use dataSource as if an abbreviation and does not have dataSourceAbbrev.
      This is inconsistent with telemetry stations. I realize that it may be difficult to change at this point. From Doug: [No Action]
2. Streamflow month and year interval volume from HydroBase does not agree with web services.
   This appears to be because the factor to convert from cfs to af units is different.
   To get the automated test to run without warnings I had to multiply the REST web service values by 1.00025216.
   I recommend that the State confirm that the conversion factor is the same for HydroBase database
   calculations and web services. From Doug: [Doug] Found that Hydrobase was using 1.984 instead of 1.9835. Have changed views in HydroBase that will take effect in the next database snapshot release. Previous releases will be slightly off. This change will likely break automated tests that I run for TSTool (when I get a new HydroBase) and will change CDSS model input slightly for larger streamflow numbers due to the number of digits used. This type of change is hard on automated testing because it requires running different tests on different versions of the database, which takes resources to configure. This is just the way it is.
3. Streamflow year interval values are water year. It would be useful to provide a calendar year service.
   I was able to compare service results by processing HydroBase monthly values into water year.
   Previous feedback indicates that some data is retrieved from third parties and I get that. From Doug: [No action]
4. Some data types are the same in climate stations and telemetry stations (if case is ignored),
   including Evap, Precip, and Solar.
   This means that TSTool time series identifiers that only use the data type would be ambiguous for Day interval.
   One solution is to use a location type of climate in the time series identifier as in
   climate:StationId.DataType.DataSource.DataInterval.
   However, It appears that climate stations do not have measType values that overlaps telemetry station.
   Therefore, I am going to try to avoid putting climate: in front of time series identifiers because TSID should be unique.
   This will have to change if ambiguity results in the future. From Doug: [No Action].
   This is mainly a TSTool convention and the State can comment on documentation when I have that done.
5. Frost dates, dateFormat does not seem to work. I tried using dateFormat=dateOnly since frost dates have precision to day. From Doug: https://dwr.state.co.us/Rest/GET/api/v2/climatedata/climatestationfrostdates/?dateFormat=dateOnly&stationNum=1 is formatting results correctly.
   My bad on this. It does work. My feedback similar to a previous comment is that in such a case it
   makes sense that the default date format would just be date (no time) since time is not used.
6. Monthly statistics:
   1. By experimentation, it appears that values are computed by using the daily values as the sample.
      That is OK but has some impacts on statistics. From Doug: [Doug] Statistics are computed using the data pulled from NOAA and CoAgMet services, which is daily.
   2. The Total monthly values do not seem to be computed for the following measType: MinTemp, MaxTemp, MeanTemp,
      SnowDepth, SnowSWE, Solar, VP, Wind. This makes sense for some because the values are instantaneous.
      However, monthly temperature total such as total of MeanTemp could be used for
      degree days when evaluating climate change.
      Total solar radiation for a month may also be useful for consumptive use modeling but I'd have to research the units more.
      Total wind run is useful to understand how windy a location is and can be used in consumptive use modeling.
      Kelley Thompson might have opinions on such things. I have the queries working and now need to write automated tests
      so I'll know soon how the web services compare to HydroBase database queries. From Doug: [Doug] These are new summations that previously have not been produced. As mentioned above, this potential request should be discussed with the CWCB/DWR Team to see if it is something we want to provide.
7. No year interval for climate stations statistics. Annual data would be useful in some cases, for example for climate
   change analysis and generally for long-term trend analysis. Water year complicates things. From Doug: [Doug] This potential request should be discussed with the CWCB/DWR Team to see if it is something we want to provide.
   1. Total precipitation.
   2. Mean temperature.
   3. Total evaporation.

Status of Resolving Issues

Overall, my questions have been answered and issues addressed. Remaining issues are related to improving documentation, which the State can do during normal maintenance. Inconsistencies in the API are unfortunately baked in at this point because people are using the API. Perhaps inconsistencies can be addressed in the future if a new version is released. The new time series that I have suggested can be discussed with the CDSS group. Enabling these will require changes to TSTool since I am currently filtering out the statistics that have all nulls in the results so as to not confuse users.

I will keep this issue open until a TSTool release is made because I may find some additional issues as I finalize the tests.

[smalers]

The following are issues found during testing. I added many tests and believe the following are unresolved. Links are provided to the current test command file because the content below may be out of date. I will publish a software release when done putting together the tests so that the State can download an run themselves. All testing was done with HydroBase_20210322. These issues may be indicative of similar issues at other stations. Although it would be possible to compare all stations in HydroBase with all stations in web services, that comparison is beyond the scope of software automated tests.

MeanTemp-Avg.Month Test fails

The following TSTool command file shows differences:

• It seems that the web services might be in degrees C rather than F,
  although doing a conversion does not seem exact so maybe the values are calculated differently?
• The first value in both time series are similar, not off by units conversion?

# Test reading an MeanTemp-Avg month interval time series from ColoradoHydroBaseRest web service using a TSID.
# - Compare the resulting time series with that retrieved from HydroBase
# - allow some missing based on database
StartLog(LogFile="Results/Test_TSID_ColoradoHydroBaseRest_CompareHydroBase_MeanTemp-Avg_Month.TSTool.log")
RemoveFile(InputFile="Results/Test_TSID_ColoradoHydroBaseRest_CompareHydroBase_MeanTemp-Avg_Month_out.dv",IfNotFound=Ignore)
# Read the same time series from the web service and HydroBase
SetInputPeriod(InputStart="2015-01",InputEnd="2018-12")
# FTC01 - FORT COLLINS
FTC01.CoAgMet.MeanTemp-Avg.Month~HydroBaseWeb
FTC01.CoAgMet.TempMean.Month~HydroBase
# Make sure that enough data are available in the test data, and some missing
CheckTimeSeriesStatistic(Statistic="NonmissingCount",CheckCriteria="<=",CheckValue1=10,IfCriteriaMet=Warn)
CheckTimeSeriesStatistic(Statistic="MissingCount",CheckCriteria=">",CheckValue1=0,IfCriteriaMet=Warn)
WriteDateValue(TSList=LastMatchingTSID,TSID="FTC01.CoAgMet.MeanTemp-Avg.Month",OutputFile="Results/Test_TSID_ColoradoHydroBaseRest_CompareHydroBase_MeanTemp-Avg_Month_out.dv",Precision=2)
CompareTimeSeries(Tolerance=".01",IfDifferent=Warn)

MeanTemp-Max.Month Test fails

I'm not sure why the following command file fails. I may not be understanding the contents of the data. Monthly statistics computed on daily statistic is confusing.

# Test reading an MeanTemp-Max month interval time series from ColoradoHydroBaseRest web service using a TSID.
# - Compare the resulting time series with that retrieved from HydroBase
# - allow some missing based on database
StartLog(LogFile="Results/Test_TSID_ColoradoHydroBaseRest_CompareHydroBase_MeanTemp-Max_Month.TSTool.log")
RemoveFile(InputFile="Results/Test_TSID_ColoradoHydroBaseRest_CompareHydroBase_MeanTemp-Max_Month_out.dv",IfNotFound=Ignore)
# Read the same time series from the web service and HydroBase
SetInputPeriod(InputStart="2015-01",InputEnd="2018-12")
# FTC01 - FORT COLLINS
FTC01.CoAgMet.MeanTemp-Max.Month~HydroBaseWeb
FTC01.CoAgMet.TempMeanMax.Month~HydroBase
# Make sure that enough data are available in the test data, and some missing
CheckTimeSeriesStatistic(Statistic="NonmissingCount",CheckCriteria="<=",CheckValue1=10,IfCriteriaMet=Warn)
CheckTimeSeriesStatistic(Statistic="MissingCount",CheckCriteria=">",CheckValue1=0,IfCriteriaMet=Warn)
WriteDateValue(TSList=LastMatchingTSID,TSID="FTC01.CoAgMet.MeanTemp-Max.Month",OutputFile="Results/Test_TSID_ColoradoHydroBaseRest_CompareHydroBase_MeanTemp-Max_Month_out.dv",Precision=2)
CompareTimeSeries(Tolerance=".01",IfDifferent=Warn)

MeanTemp-Min.Month Test fails

I'm not sure why the following command file fails. I may not be understanding the contents of the data. Monthly statistics computed on daily statistic is confusing.

# Test reading an MeanTemp-Min month interval time series from ColoradoHydroBaseRest web service using a TSID.
# - Compare the resulting time series with that retrieved from HydroBase
# - allow some missing based on database
StartLog(LogFile="Results/Test_TSID_ColoradoHydroBaseRest_CompareHydroBase_MeanTemp-Min_Month.TSTool.log")
RemoveFile(InputFile="Results/Test_TSID_ColoradoHydroBaseRest_CompareHydroBase_MeanTemp-Min_Month_out.dv",IfNotFound=Ignore)
# Read the same time series from the web service and HydroBase
SetInputPeriod(InputStart="2015-01",InputEnd="2018-12")
# FTC01 - FORT COLLINS
FTC01.CoAgMet.MeanTemp-Min.Month~HydroBaseWeb
FTC01.CoAgMet.TempMeanMin.Month~HydroBase
# Make sure that enough data are available in the test data, and some missing
CheckTimeSeriesStatistic(Statistic="NonmissingCount",CheckCriteria="<=",CheckValue1=10,IfCriteriaMet=Warn)
CheckTimeSeriesStatistic(Statistic="MissingCount",CheckCriteria=">",CheckValue1=0,IfCriteriaMet=Warn)
WriteDateValue(TSList=LastMatchingTSID,TSID="FTC01.CoAgMet.MeanTemp-Min.Month",OutputFile="Results/Test_TSID_ColoradoHydroBaseRest_CompareHydroBase_MeanTemp-Min_Month_out.dv",Precision=2)
CompareTimeSeries(Tolerance=".01",IfDifferent=Warn)

Snow.Day Test Fails

The following test fails. The HydroBase database has a much shorter period and values are slightly different, although it has more recent data and web services does not.

# Test reading an Snow day interval time series from ColoradoHydroBaseRest web service using a TSID.
# - Compare the resulting time series with that retrieved from HydroBase
# - allow a high number of missing based on database, due to winter, etc.
StartLog(LogFile="Results/Test_TSID_ColoradoHydroBaseRest_CompareHydroBase_Snow_Day.TSTool.log")
RemoveFile(InputFile="Results/Test_TSID_ColoradoHydroBaseRest_CompareHydroBase_Snow_Day_out.dv",IfNotFound=Ignore)
# Read the same time series from the web service and HydroBase
SetInputPeriod(InputStart="1949-04-06",InputEnd="2018-12-31")
# USC00053005 - FORT COLLINS
USC00053005.NOAA.Snow.Day~HydroBaseWeb
USC00053005.NOAA.Snow.Day~HydroBase
# Make sure that enough data are available in the test data, and some missing
CheckTimeSeriesStatistic(Statistic="NonmissingCount",CheckCriteria="<=",CheckValue1=10,IfCriteriaMet=Warn)
CheckTimeSeriesStatistic(Statistic="MissingCount",CheckCriteria=">",CheckValue1=11501,IfCriteriaMet=Warn)
WriteDateValue(TSList=LastMatchingTSID,TSID="USC00053005.NOAA.Snow.Day",OutputFile="Results/Test_TSID_ColoradoHydroBaseRest_CompareHydroBase_Snow_Day_out.dv",Precision=2)
CompareTimeSeries(Tolerance=".01",IfDifferent=Warn)

Snow-Max.Month Test Fails

The Snow-Total.Month test also fails.

The following test fails. The HydroBase database and web service values for last part of the time series are different. I may just need to use a newer version of HydroBase that has updated data.

# Test reading a Snow-Max month interval time series from ColoradoHydroBaseRest web service using a TSID.
# - compare the resulting time series with that retrieved from HydroBase
# - HydroBase does not include the monthly statistic so compute from daily and then compare
StartLog(LogFile="Results/Test_TSID_ColoradoHydroBaseRest_CompareHydroBase_Snow-Max_Month.TSTool.log")
RemoveFile(InputFile="Results/Test_TSID_ColoradoHydroBaseRest_CompareHydroBase_Snow-Max_Month_out.dv",IfNotFound=Ignore)
# Read the same time series from the web service and HydroBase
SetInputPeriod(InputStart="2014-08",InputEnd="2018-05")
# USC00053005 - FORT COLLINS
USC00053005.NOAA.Snow-Max.Month~HydroBaseWeb
USC00053005.NOAA.Snow.Day~HydroBase
SetInputPeriod(InputStart="2014-08-01",InputEnd="2018-05-31")
NewStatisticMonthTimeSeries(TSID="USC00053005.NOAA.Snow.Day",Alias="USC00053005-HydroBase-Month",NewTSID="USC00053005..Snow-Max.Month",Statistic=Max)
# Make sure that enough data are available in the test data, and some missing.
CheckTimeSeriesStatistic(Statistic="NonmissingCount",CheckCriteria="<=",CheckValue1=10,IfCriteriaMet=Warn)
CheckTimeSeriesStatistic(TSList=AllMatchingTSID,TSID="*Month",Statistic="MissingCount",CheckCriteria=">",CheckValue1=0,IfCriteriaMet=Warn)
WriteDateValue(TSList=LastMatchingTSID,TSID="USC00053005.NOAA.Snow-Max.Month",OutputFile="Results/Test_TSID_ColoradoHydroBaseRest_CompareHydroBase_Snow-Max_Day_out.dv",Precision=2)
CompareTimeSeries(TSID1="USC00053005.NOAA.Snow-Max.Month",TSID2="USC00053005-HydroBase-Month",Tolerance=".01",IfDifferent=Warn)

SnowDepth.Day Test fails

The following test fails. The patterns are similar but numbers are different. Do I need an updated HydroBase?

# Test reading a SnowDepth day interval time series from ColoradoHydroBaseRest web service using a TSID.
# - compare the resulting time series with that retrieved from HydroBase
# - allow a high number of missing based on database
StartLog(LogFile="Results/Test_TSID_ColoradoHydroBaseRest_CompareHydroBase_SnowDepth_Day.TSTool.log")
RemoveFile(InputFile="Results/Test_TSID_ColoradoHydroBaseRest_CompareHydroBase_SnowDepth_Day_out.dv",IfNotFound=Ignore)
# Read the same time series from the web service and HydroBase
SetInputPeriod(InputStart="1949-04-06",InputEnd="2018-12-31")
# USC00053005 - FORT COLLINS
USC00053005.NOAA.SnowDepth.Day~HydroBaseWeb
USC00053005.NOAA.SnowCourseDepth.Day~HydroBase
# Make sure that enough data are available in the test data, and some missing
CheckTimeSeriesStatistic(Statistic="NonmissingCount",CheckCriteria="<=",CheckValue1=10,IfCriteriaMet=Warn)
CheckTimeSeriesStatistic(Statistic="MissingCount",CheckCriteria=">",CheckValue1=11501,IfCriteriaMet=Warn)
WriteDateValue(TSList=LastMatchingTSID,TSID="USC00053005.NOAA.SnowDepth.Day",OutputFile="Results/Test_TSID_ColoradoHydroBaseRest_CompareHydroBase_SnowDepth_Day_out.dv",Precision=2)
CompareTimeSeries(Tolerance=".01",IfDifferent=Warn)

SnowDepth-Avg.Month Test Fails

The following test fails, likely because the daily data test fails.

# Test reading a SnowDepth-Avg month interval time series from ColoradoHydroBaseRest web service using a TSID.
# - compare the resulting time series with that retrieved from HydroBase
# - HydroBase does not include the monthly statistic so compute from daily and then compare
StartLog(LogFile="Results/Test_TSID_ColoradoHydroBaseRest_CompareHydroBase_SnowDepth-Avg_Month.TSTool.log")
RemoveFile(InputFile="Results/Test_TSID_ColoradoHydroBaseRest_CompareHydroBase_SnowDepth-Avg_Month_out.dv",IfNotFound=Ignore)
# Read the same time series from the web service and HydroBase.
SetInputPeriod(InputStart="2014-08",InputEnd="2018-05")
# USC00053005 - FORT COLLINS
USC00053005.NOAA.SnowDepth-Avg.Month~HydroBaseWeb
USC00053005.NOAA.SnowCourseDepth.Day~HydroBase
SetInputPeriod(InputStart="2014-08-01",InputEnd="2018-05-31")
NewStatisticMonthTimeSeries(TSID="USC00053005.NOAA.SnowCourseDepth.Day",Alias="USC00053005-HydroBase-Month",NewTSID="USC00053005..SnowCourseDepth-Avg.Month",Statistic=Mean)
# Make sure that enough data are available in the test data, and some missing.
CheckTimeSeriesStatistic(Statistic="NonmissingCount",CheckCriteria="<=",CheckValue1=10,IfCriteriaMet=Warn)
CheckTimeSeriesStatistic(TSList=AllMatchingTSID,TSID="*Month",Statistic="MissingCount",CheckCriteria=">",CheckValue1=0,IfCriteriaMet=Warn)
WriteDateValue(TSList=LastMatchingTSID,TSID="USC00053005.NOAA.SnowDepth-Avg.Month",OutputFile="Results/Test_TSID_ColoradoHydroBaseRest_CompareHydroBase_SnowDepth-Avg_Day_out.dv",Precision=2)
CompareTimeSeries(TSID1="USC00053005.NOAA.SnowDepth-Avg.Month",TSID2="USC00053005-HydroBase-Month",Tolerance=".01",IfDifferent=Warn)

SnowDepth-Max.Month Test Fails

The following test fails, likely because the daily data test fails.

Note that the minimum test passes but that is because all of the values are zero.

# Test reading a SnowDepth-Max month interval time series from ColoradoHydroBaseRest web service using a TSID.
# - compare the resulting time series with that retrieved from HydroBase
# - HydroBase does not include the monthly statistic so compute from daily and then compare
StartLog(LogFile="Results/Test_TSID_ColoradoHydroBaseRest_CompareHydroBase_SnowDepth-Max_Month.TSTool.log")
RemoveFile(InputFile="Results/Test_TSID_ColoradoHydroBaseRest_CompareHydroBase_SnowDepth-Max_Month_out.dv",IfNotFound=Ignore)
# Read the same time series from the web service and HydroBase
SetInputPeriod(InputStart="2014-08",InputEnd="2018-05")
# USC00053005 - FORT COLLINS
USC00053005.NOAA.SnowDepth-Max.Month~HydroBaseWeb
USC00053005.NOAA.SnowCourseDepth.Day~HydroBase
SetInputPeriod(InputStart="2014-08-01",InputEnd="2018-05-31")
NewStatisticMonthTimeSeries(TSID="USC00053005.NOAA.SnowCourseDepth.Day",Alias="USC00053005-HydroBase-Month",NewTSID="USC00053005..SnowCourseDepth-Max.Month",Statistic=Max)
# Make sure that enough data are available in the test data, and some missing.
CheckTimeSeriesStatistic(Statistic="NonmissingCount",CheckCriteria="<=",CheckValue1=10,IfCriteriaMet=Warn)
CheckTimeSeriesStatistic(TSList=AllMatchingTSID,TSID="*Month",Statistic="MissingCount",CheckCriteria=">",CheckValue1=0,IfCriteriaMet=Warn)
WriteDateValue(TSList=LastMatchingTSID,TSID="USC00053005.NOAA.SnowDepth-Max.Month",OutputFile="Results/Test_TSID_ColoradoHydroBaseRest_CompareHydroBase_SnowDepth-Max_Day_out.dv",Precision=2)
CompareTimeSeries(TSID1="USC00053005.NOAA.SnowDepth-Max.Month",TSID2="USC00053005-HydroBase-Month",Tolerance=".01",IfDifferent=Warn)

Solar.Day Test fails.

The following test fails.
The HydroBase database and web service values match during the start and end of the period but the middle is totally different. This is odd because other tests for statistics based on the daily data pass.

# Test reading a Solar day interval time series from ColoradoHydroBaseRest web service using a TSID.
# - compare the resulting time series with that retrieved from HydroBase
# - allow a high number of missing based on database
StartLog(LogFile="Results/Test_TSID_ColoradoHydroBaseRest_CompareHydroBase_Solar_Day.TSTool.log")
RemoveFile(InputFile="Results/Test_TSID_ColoradoHydroBaseRest_CompareHydroBase_Solar_Day_out.dv",IfNotFound=Ignore)
# Read the same time series from the web service and HydroBase
SetInputPeriod(InputStart="2015-01-01",InputEnd="2020-07-20")
# USC00053005 - FORT COLLINS
FCL01.CoAgMet.Solar.Day~HydroBaseWeb
FCL01.CoAgMet.Solar.Day~HydroBase
# Make sure that enough data are available in the test data, and some missing
CheckTimeSeriesStatistic(Statistic="NonmissingCount",CheckCriteria="<=",CheckValue1=10,IfCriteriaMet=Warn)
CheckTimeSeriesStatistic(Statistic="MissingCount",CheckCriteria=">",CheckValue1=153,IfCriteriaMet=Warn)
WriteDateValue(TSList=LastMatchingTSID,TSID="FCL01.CoAgMet.Solar.Day",OutputFile="Results/Test_TSID_ColoradoHydroBaseRest_CompareHydroBase_Solar_Day_out.dv",Precision=2)
CompareTimeSeries(Tolerance=".1",IfDifferent=Warn)

Wind.Day Test fails

The following test fails. The HydroBase database and web service values match during the start and end of the period but the middle is totally different. This is odd because other tests for statistics based on the daily data pass.

# Test reading a Wind day interval time series from ColoradoHydroBaseRest web service using a TSID.
# - Compare the resulting time series with that retrieved from HydroBase
# - allow a high number of missing based on database, due to winter, etc.
StartLog(LogFile="Results/Test_TSID_ColoradoHydroBaseRest_CompareHydroBase_Wind_Day.TSTool.log")
RemoveFile(InputFile="Results/Test_TSID_ColoradoHydroBaseRest_CompareHydroBase_Wind_Day_out.dv",IfNotFound=Ignore)
# Read the same time series from the web service and HydroBase
SetInputPeriod(InputStart="2015-01-01",InputEnd="2020-07-20")
# USC00053005 - FORT COLLINS
FCL01.CoAgMet.Wind.Day~HydroBaseWeb
FCL01.CoAgMet.Wind.Day~HydroBase
# Make sure that enough data are available in the test data, and some missing
CheckTimeSeriesStatistic(Statistic="NonmissingCount",CheckCriteria="<=",CheckValue1=10,IfCriteriaMet=Warn)
CheckTimeSeriesStatistic(Statistic="MissingCount",CheckCriteria=">",CheckValue1=153,IfCriteriaMet=Warn)
WriteDateValue(TSList=LastMatchingTSID,TSID="FCL01.CoAgMet.Wind.Day",OutputFile="Results/Test_TSID_ColoradoHydroBaseRest_CompareHydroBase_Wind_Day_out.dv",Precision=2)
CompareTimeSeries(Tolerance=".1",IfDifferent=Warn)