Reproduce bug #294
Merged
Reproduce bug #294
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… was not properly reset, then the results of models that should be identical would diverge.
…eproducible outcomes.
amoodie
force-pushed
the
reproduce_bug
branch
from
da51f76 to
0cdc555
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## develop #294 +/- ##
===========================================
+ Coverage 79.00% 79.12% +0.11%
===========================================
Files 12 12
Lines 2643 2639 -4
===========================================
Hits 2088 2088
+ Misses 555 551 -4 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
amoodie
force-pushed
the
reproduce_bug
branch
from
a7d31c3 to
3e3829f
Compare
Member
Author
|
tests are passing; failure is due to existing links in docs, and some install quirk |
elbeejay
approved these changes
Oct 22, 2025
Member
elbeejay
left a comment
elbeejay
left a comment
There was a problem hiding this comment.
Looks good, appreciate you resolving it!
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR fixes an issue with reproducibility of runs with different
save_dt#205.I finally traced this issue down to the seed not being set correctly on parallel jobs. This impacts the reproducibility of runs that were executed in parallel via the preprocessor. The specified seed may not match the parent-inherited seed in all cases. This is very unfortunate, but cannot be fixed in the past.
The fix seems to be resetting the seed explicitly in any job (serial or parallel) when that job is definitively on the thread it will run on, within the preprocesser
_SerialJobor_ParallelJob.@elbeejay I believe there were some problems with the tests demonstrating this bug in your original post, but they also did clearly reproduce the issue. I think the first failed in part because it expected the final beds to be the same, but the ModelB was only saved once at the initial time, so the final times would not be the same. I think the second passed because the jobs were run in serial. In any case, I think the new tests I have added show that the issue with
save_dtbeing different and unreproducible results, is resolved.@elbeejay sorry for the delay on merging #291 , I was afraid this old bug could impact it somehow and wanted to get this sorted first. After we review this, I'd like to get your PR there merged.
fixes #205