Cooldown state rework #455
Open
+1,817
−1,196
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
entropidelic
reviewed
Jan 9, 2026
entropidelic
reviewed
Jan 9, 2026
dsocolobsky
reviewed
Jan 9, 2026
| hub_repo | ||
| ) | ||
| let Model::LLM(LLM { checkpoint, .. }) = &self.coordinator_state.model; | ||
| if !self.skip_upload_check { |
Contributor
There was a problem hiding this comment.
maybe we can invert the check and return early
Contributor
Author
There was a problem hiding this comment.
We have to execute the rest of the code in that function, so I don’t think we can return early there, unless you meant something else?
Comment on lines
+211
to
+213
| if skip_upload { | ||
| info!("Skipping checkpoint save and upload (skip_upload flag is set)"); | ||
| checkpoint_completed.store(true, Ordering::SeqCst); |
Contributor
There was a problem hiding this comment.
you can return Ok(evals) early and you don't need to have an else branch here
shared/data-provider/src/errors.rs
Outdated
Comment on lines
16
to
17
| //#[error("GCS operation failed: {0}")] | ||
| //GcsStorage(#[from] google_cloud_storage::client::Error), |
Contributor
There was a problem hiding this comment.
should we delete this?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR contains several changes to client arguments, dependencies, and the general cooldown state logic. Here’s a summary of the changes made in this PR:
SOLANA_MAX_NUM_CHECKPOINTERS, which is currently 16) is deterministically selected as checkpointers using a seeded shuffle algorithm based on the round’s random seed.cooldown_timebefore moving on.--hub-repoand--gcs-bucketflags were removed from the training arguments. Checkpoint destinations are now derived from the coordinator configuration, and there is a single repo or bucket that contains all the checkpoints for the run.--skip-checkpoint-upload, for testing purposes. This avoids uploading and saving checkpoints locally and should not be used outside testing environments.google-cloud-storagecrate to version 1.6.SCRATCH_DIRin order to work correctly when using GCP checkpoints.