Fix/scheduling/sign

[Flix6x]

Description

• Fix the sign for schedules.
• Increment the version of all schedulers?
• Added changelog item in documentation/changelog.rst

Look & Feel

Work in progress: before / after.

How to test

This needs a plan for testing on multiple servers.

Further Improvements

• Database migration
• Alternatively (?) a CLI script for manual migration, that let's you do something like:
  update timed_belief set event_value = -event_value where sensor_id=1 and source_id in (1, 2, 3) and event_start>='2022-12-24T14:00+02:00' and event_start<'2025-10-01T15:00+02:00;
  Maybe use sensor.sources to find out the sources of the "scheduler" type.

Related Items

Closes #1345.

[read-the-docs-community]

Documentation build overview

📚 flexmeasures | 🛠️ Build #31073142 | 📁 Comparing 5be69a3 against latest (64e6040)

🔍 Preview build

Show files changed (4 files in total): 📝 4 modified | ➕ 0 added | ➖ 0 deleted

File	Status
changelog.html	📝 modified
genindex.html	📝 modified
_autosummary/flexmeasures.utils.config_defaults.html	📝 modified
api/v3_0.html	📝 modified

[nhoening]

Okay, so this needs a migration script (as part of the main TODOs), does it not?
I tested this with schedules made on main, and they do not show any different in the graphs.

[nhoening]

A test plan:

1. Test it on the HEMS script - checking if schedules that were made without it look good, then re-creating everything with it.
2. Test with local dump of ems_production, with old & new schedules as well
3. Test on ems_staging (maybe quickly applying the diff to the one web server and worker) against a real case (V2GLiberty), including the effect this change has on schedules delivered to the assets.
4. Also test on our simulation code internally)

[nhoening]

I ran step 1: schedules look good when the HEMS script is run with that PR activated.

[Flix6x]

I ran step 3: as expected, clients without the "consumption_is_positive" attribute set are unaffected by the changes in this PR. After the attribute is added, clients remain unaffected, while any new schedules from that point on will respect the intended sign switch. Old schedule data still needs to be flipped. To facilitate that operation, we floated the idea of incrementing the version of all schedulers, to force the creation of a new data source ID.

The test excluded clients with the "consumption_is_positive" attribute set (to either true or false). As the suggested workaround was to (counterintuitively) set the "consumption_is_positive" attribute to false, we do have clients with this very setting. Other hosts may have implemented this workaround, too. For that reason, the attribute should be flipped, too, as part of this PR. I would add a warning in the db migration about which assets/sensors are affected, but I believe an automated flip would always be preferred to keep the status quo in terms of the sign of new schedules (which in these cases was already correct - again, by setting the attribute purposely incorrectly). We would then support the following cases:

1. No attribute set: with this PR, nothing changes, but the user can now set the "consumption_is_positive" attribute do get the desired behaviour without any side effects.
2. Attribute was set to False: with this PR, the attribute will flip to True, so otherwise nothing changes.
3. Attribute was set to True: not sure about this case yet.

[nhoening]

Thanks for doing this sizable step! Can you indicate what the situation in the V2GLiberty power sensors actually was w.r.t. "consumption_is_positive" attribute?

I got a question about signs from a hoster just this afternoon (he wondered about the sign of solar data in the tutorial). Maybe this PR could also help clear up where we'd expect which one of our two perspectives.

We only have some API documentation saying we take the Prosumer/Asset perspective in schedules/ the API.

This PR also switches between these two perspectives - and that is why there are two places where it changes the if statements that might apply * -1

We don't even have clear names and descriptions yet, I believe this is the state of things (correct me if I'm wrong, the more I think the more I believe I might be):

• One perspective is the "inside": what we store in the DB, and what is also represented as such in plots. Here, "consumption is negative unless specified explicitly", production is positive.
• The other is "outside": what we give to others - here "positive values denote consumption", negative is production.
  • Apparently this only holds for schedules, not for forecasts?
  • I forgot what our reason was why we chose to switch this for the outside - USEF?

I'll gladly have a quick session on this so we can give others one clear section in the docs (if I and the host haven't missed it).

[nhoening]

(I will gladly write the docs section if needed)

[Flix6x]

Thanks for doing this sizable step! Can you indicate what the situation in the V2GLiberty power sensors actually was w.r.t. "consumption_is_positive" attribute?

They did not have it set.

I got a question about signs from a hoster just this afternoon (he wondered about the sign of solar data in the tutorial). Maybe this PR could also help clear up where we'd expect which one of our two perspectives.

We only have some API documentation saying we take the Prosumer/Asset perspective in schedules/ the API.

This PR also switches between these two perspectives - and that is why there are two places where it changes the if statements that might apply * -1

I recommend an extra step: checking the tutorial results against this PR. Maybe we now actually need to set the attribute for a couple (or all) of the sensors.

We don't even have clear names and descriptions yet, I believe this is the state of things (correct me if I'm wrong, the more I think the more I believe I might be):

• One perspective is the "inside": what we store in the DB, and what is also represented as such in plots. Here, "consumption is negative unless specified explicitly", production is positive.

• The other is "outside": what we give to others - here "positive values denote consumption", negative is production.

  • Apparently this only holds for schedules, not for forecasts?
  • I forgot what our reason was why we chose to switch this for the outside - USEF?

This should be correct, including the USEF reason, as far as I remember and according to our own docs (from your link). Our new forecast endpoint is not USEF-like, and we used to have separate endpoints for getting and posting Prognoses, as described by USEF, which did follow USEF sign convention, but these have been sunset years ago.

I'll gladly have a quick session on this so we can give others one clear section in the docs (if I and the host haven't missed it).

I don't think this can be a quick session, at least not yet. I am still in the process of scoping how much of a breaking change this PR represents, and trying to document that on this PR. Two extra things to be mindful of:

1. In data.models.planning.utils.get_power_values we also use the "consumption_is_positive" attribute (and without the bug), which implies that we should not flip the attribute when migrating for any power sensor used as an inflexible device, which is not trivial to identify (at least not as a dev, without being familiar with the host's data).
2. I encountered cost reporters for V2GLiberty whose config seems to work around the bug using "loss_is_positive": false. As noted above, the power sensor serving as input did not have the attribute set. I need to further investigate the implications.

[nhoening]

Then we should discuss the documentation as a last step of this endeavor.

[nhoening]

Is it an option for the future that we stop flipping the sign for schedules, to simplify how we treat data? I feel it has more costs than benefits.

Maybe for one version or so we can have a setting for hosts to keep it the old way for them.

[Flix6x]

Is it an option for the future that we stop flipping the sign for schedules, to simplify how we treat data? I feel it has more costs than benefits.

Maybe for one version or so we can have a setting for hosts to keep it the old way for them.

I don't see how this would work. The scheduler needs to have some concept of what is up and down when it applies prices, because it is trying to minimize costs.