Skip to content

Conversation

@pefontana
Copy link
Contributor

@pefontana pefontana commented Jan 5, 2026

Implement GCS Checkpoint to download model from Google Cloud Storage

Now, in the run config, we can set the model URL as a Google Cloud Storage bucket:
The GCS checkpoint is completely decoupled from Hugging Face one, because in the mid term the idea is to remove the HF checkpoint and just leave the GCS one

  [model.LLM.checkpoint.Gcs]
  bucket = "<BUCKET_NAME>"
  prefix = "<BUCKET_FOLDER>"

To access the bucket, pass the credentials as an env var:

GOOGLE_APPLICATION_CREDENTIALS=<CREDENTIAL_PATH> just start-training-localnet-light-client

Also supports evaluation with models stored in GCS:

GOOGLE_APPLICATION_CREDENTIALS=<CREDENTIAL_PATH> \
    cargo run -p psyche-eval --example evaluate -- \
    --gcs-bucket "<BUCKET_NAME>" \
    --gcs-prefix "<BUCKET_FOLDER>" \
    --tasks "arc_easy"

Summary

  • Implement Checkpoint::Gcs(GcsRepo)
  • Implement Checkpoint::GcsP2P(gcs_repo) to falls back to GCS if P2P fails
  • Implement pub async fn download_model_from_gcs_async() to download model from GCS
  • Add traces to evaluation crate to track model download

Includes

@pefontana pefontana changed the title Download model from google cloud storage Implement Gcs Checkpoint: Download model from Google Cloud Storage Jan 9, 2026
@pefontana pefontana marked this pull request as ready for review January 9, 2026 16:02
@pefontana pefontana changed the title Implement Gcs Checkpoint: Download model from Google Cloud Storage Implement GCS Checkpoint: Download model from Google Cloud Storage Jan 9, 2026
@jquesnelle jquesnelle added contract change change to on-chain contract logic storage change change to on-chain storage layout labels Jan 13, 2026
@pefontana pefontana removed contract change change to on-chain contract logic storage change change to on-chain storage layout labels Jan 14, 2026
@pefontana pefontana changed the title Implement GCS Checkpoint: Download model from Google Cloud Storage Implement GCS Checkpoint Jan 15, 2026
@jquesnelle jquesnelle added storage change change to on-chain storage layout contract change change to on-chain contract logic labels Jan 22, 2026
@pefontana
Copy link
Contributor Author

@jquesnelle @crypto-vincent @arilotter
I took a look at the contract/storage change tags and wanted to confirm my understanding.

From what I see, we'd need to redeploy the program since Checkpoint now accepts the new Gcs and P2PGcs variants. But
for storage, since we're just adding enum variants at the end and GcsRepo has the same size as HubRepo (both use two
FixedString fields), it should be backward compatible with existing accounts, right? The memnet_coordinator_data_layout test passes, so I don't think we need any migration

So can we remove the storage change tag?

@arilotter
Copy link
Collaborator

Yeah, i agree that it should be marked as a contract change only, and that the storage test passing is evidence that it hasn't changed in representation.

@jquesnelle jquesnelle removed the storage change change to on-chain storage layout label Jan 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

contract change change to on-chain contract logic

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants