Skip to content

feat: show statistics of the job queueing system #1838

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
Flix6x wants to merge 22 commits into
base: main
Choose a base branch
from
Open

feat: show statistics of the job queueing system #1838

Flix6x wants to merge 22 commits into from

Conversation

@Flix6x
Copy link
Contributor

@Flix6x Flix6x commented Nov 28, 2025 •
edited
Loading

Description

  • flexmeasures jobs stats
  • Added changelog item in documentation/changelog.rst
  • Added changelog item in documentation/cli/change_log.rst

Look & Feel

Estimating arrival rates using a 60-minute window...

Overall: k=2, L=0.40, ρ=20%

Queue          λ (/s)    Lq    Ls    Wq (s)    Ws (s)    W (s)
-----------  --------  ----  ----  --------  --------  -------
forecasting    0.0033     0  0.08         0     22.62    22.62
scheduling     0.015      0  0.33         0     21.81    21.81

How to test

Run flexmeasures jobs stats (preferably with an operational redis DB in your flexmeasures.cfg settings).

Further Improvements

  • Save the results as sensor data (maybe automatically setting up a custom asset to hold all the sensors)

@Flix6x
feat: poc for showing average waiting time per queue
f9a16ca 
Signed-off-by: F.N. Claessen <felix@seita.nl>
@read-the-docs-community
Copy link

read-the-docs-community bot commented Nov 28, 2025 •
edited
Loading

Documentation build overview

📚 flexmeasures | 🛠️ Build #31081039 | 📁 Comparing 5f65f84 against latest (64e6040)


🔍 Preview build

Show files changed (6 files in total): 📝 6 modified | ➕ 0 added | ➖ 0 deleted
File Status
changelog.html 📝 modified
genindex.html 📝 modified
_autosummary/flexmeasures.cli.jobs.html 📝 modified
_autosummary/flexmeasures.utils.config_defaults.html 📝 modified
api/v3_0.html 📝 modified
cli/change_log.html 📝 modified

@Flix6x
feat: report servicing time
63a0274 
Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x
feat: report on capacity utilization
abe03a3 
Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x
docs: update docstring
6df87d3 
Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x
style: tabulate console output
6d5bec8 
Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x
feat: include waiting jobs in overall stats, so we report capacity re…
358092a 
…quirement rather than capacity utilization (otherwise, we could never see ρ > 100%)

Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x
style: revise orderings so that the console output shows stats in rev…
44644fb 
…erse order with respect to the help string

Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x
feat: use 1-hour default window instead of 10 minutes
c7975ee 
Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x
Merge remote-tracking branch 'refs/remotes/origin/main' into feat/ave…
94f738b 
…rage-waiting-time
@Flix6x
docs: changelog entry
b37e434 
Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x Flix6x changed the title feat: PoC for showing average waiting time per queue feat: show statistics of the job queueing system Dec 17, 2025
@Flix6x Flix6x marked this pull request as ready for review December 17, 2025 10:28
@Flix6x Flix6x self-assigned this Dec 17, 2025
@Flix6x Flix6x requested a review from nhoening December 17, 2025 10:29
nhoening
nhoening requested changes Jan 18, 2026
View reviewed changes
Copy link
Contributor

@nhoening nhoening left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

Seems to work well. Most comments are about usability.

flexmeasures/cli/jobs.py
\b
Stats overall:
- ρ = average capacity requirement (consider scaling up the number of workers when close to or higher than 100%)
Copy link
Contributor

@nhoening nhoening Jan 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to use these generic variable names (p, K, L)? They are not used on the Wikipedia page for Little's-law. (L is being used, but you are using it now for both workers and jobs, it seems jobs is the right one)

They appear to the user in the important line "Overall: [etc]".

We can call them "capacity utilised", "workers required" and "workers available"

Copy link
Contributor Author

@Flix6x Flix6x Jan 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good comment.

  • L should indeed be foremost defined in terms of number of jobs. I made the implicit assumption that our goal is zero waiting time, in which case it would also be equal to the workers required.
  • rho (not p) is pretty standard in the literature.
  • k was wrong, the canonical variable name is m

Copy link
Contributor

@nhoening nhoening Jan 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a PhD and still cannot recall "standard literature" variable names and their meaning, then apply them to this context. Do you expect all FlexMeasures hosts have Master's degrees?

Why the aversion to making this user-friendly? (adding a few words is much friendlier than requiring all first users to call --help).

Currently, these words might be "capacity utilised", "jobs in system" and "workers available".

@nhoening nhoening added this to the 0.31.0 milestone Jan 18, 2026
Flix6x and others added 10 commits January 19, 2026 11:30
@Flix6x @nhoening
docs: make clearer that we estimate from history, from where the data…
dc17810 
… comes, and where to go for deeper understanding.

Co-authored-by: Nicolas Höning <nicolas@seita.nl>
Signed-off-by: Felix Claessen <30658763+Flix6x@users.noreply.github.com>
@Flix6x @nhoening
docs: more in line with the first message that is being sent
997972f 
Co-authored-by: Nicolas Höning <nicolas@seita.nl>
Signed-off-by: Felix Claessen <30658763+Flix6x@users.noreply.github.com>
@Flix6x @nhoening
docs: more explicit explanation
960354b 
Co-authored-by: Nicolas Höning <nicolas@seita.nl>
Signed-off-by: Felix Claessen <30658763+Flix6x@users.noreply.github.com>
@Flix6x
style: black
25d5e29 
Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x
refactor: get_all_registries
2826716 
Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x
docs: reformulate L in terms of numbers of jobs first
12c4ad9 
Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x
docs: introduce queue classification
92bf705 
Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x
fix: the canonical variable for number of servers is m, not k
ce7515c 
Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x
docs: explain that the current method is a snapshot estimate and add …
9ae7a6d 
…notes to generate a better estimate and a risk averse estimate of the capacity requirement

Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x
docs: CLI changelog entry
164fb21 
Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x Flix6x requested a review from nhoening January 19, 2026 23:39
@Flix6x
Copy link
Contributor Author

Flix6x commented Jan 19, 2026

(I'll resolve the changelog conflicts soon.)

@Flix6x
Merge remote-tracking branch 'refs/remotes/origin/main' into feat/ave…
927b9ff 
…rage-waiting-time

# Conflicts:
#	documentation/changelog.rst
#	documentation/cli/change_log.rst
nhoening
nhoening requested changes Jan 20, 2026
View reviewed changes
Copy link
Contributor

@nhoening nhoening left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for applying improvements, I only don't agree with one point.

flexmeasures/cli/jobs.py
\b
Stats overall:
- ρ = average capacity requirement (consider scaling up the number of workers when close to or higher than 100%)
Copy link
Contributor

@nhoening nhoening Jan 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a PhD and still cannot recall "standard literature" variable names and their meaning, then apply them to this context. Do you expect all FlexMeasures hosts have Master's degrees?

Why the aversion to making this user-friendly? (adding a few words is much friendlier than requiring all first users to call --help).

Currently, these words might be "capacity utilised", "jobs in system" and "workers available".

@Flix6x @nhoening
docs: remove orphaned parenthesis
5f65f84 
Co-authored-by: Nicolas Höning <nicolas@seita.nl>
Signed-off-by: Felix Claessen <30658763+Flix6x@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@nhoening nhoening nhoening requested changes

Requested changes must be addressed to merge this pull request.

Assignees

@Flix6x Flix6x

Labels

jobs monitoring

Projects

None yet

Milestone

0.31.0

Development

Successfully merging this pull request may close these issues.

3 participants

@Flix6x @nhoening