Add resumable upload support to Walrus high-level APIs#952
Draft
hayes-mysten wants to merge 3 commits intomainfrom
Draft
Add resumable upload support to Walrus high-level APIs#952hayes-mysten wants to merge 3 commits intomainfrom
hayes-mysten wants to merge 3 commits intomainfrom
Conversation
Add crash-resumable upload flows for production pipelines processing large volumes of files. Key changes: - Unified WriteBlobStep discriminated union used as return values from flow methods, async iterator yields, onStep callbacks, and resume input - Flow resume: writeBlobFlow() and writeFilesFlow() accept a resume option with prior step state to skip completed work - run() async iterator on flows for convenient full-pipeline execution - executeRegister()/executeCertify() convenience methods on flows - Smart sliver deduplication matching the Rust SDK's three-gate strategy (metadata status check, size heuristic, per-sliver status check) - getSliverStatus()/getMetadataStatus() storage node client methods - getStorageConfirmations()/getBlobObject() client helper methods - onStep/resume on writeBlob() and writeFiles() (now use flow internally) - computeBlobMetadata() accepts optional nonce for deterministic resume Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
hayes-mysten
temporarily deployed
to
sui-typescript-aws-kms-test-env
GitHub Actions
Inactive
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
hayes-mysten
temporarily deployed
to
sui-typescript-aws-kms-test-env
GitHub Actions
Inactive
Add docs site page at walrus/resumable-uploads covering: - run() async iterator with checkpoint saving - writeBlob() onStep callback and resume - Manual step-by-step flow with executeRegister/executeCertify - Step result types reference table - Helper methods for advanced resume - Smart sliver deduplication explanation Also update the package README with the same content. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
hayes-mysten
force-pushed
the
ml/walrus-resumable-uploads
branch
from
1e40bda to
7408feb
Compare
hayes-mysten
had a problem deploying
to
sui-typescript-aws-kms-test-env
GitHub Actions
Failure
hayes-mysten
temporarily deployed
to
sui-typescript-aws-kms-test-env
GitHub Actions
Inactive
Integrate run(), executeRegister/executeCertify, resume, and onStep into the existing flow and writeBlob sections rather than a separate page. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
hayes-mysten
force-pushed
the
ml/walrus-resumable-uploads
branch
from
60bd018 to
051a4ff
Compare
hayes-mysten
temporarily deployed
to
sui-typescript-aws-kms-test-env
GitHub Actions
Inactive
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Add crash-resumable upload flows for production pipelines processing large volumes of files. Motivated by partner feedback (Tatum, ingesting 2.75TB / 400k+ files) that the high-level SDK flows are not crash-resumable in practice.
Key changes:
Unified
WriteBlobSteptype: Discriminated union (encoded|registered|uploaded|certified) used as return values from flow methods, async iterator yields,onStepcallbacks, and resume input. Each step result is directly saveable to a database and passable asresume.Flow resume:
writeBlobFlow()andwriteFilesFlow()accept aresumeoption with prior step state. When resuming, completed steps are skipped and blob consistency is validated (blobId checked against re-encoded content).run()async iterator: New method on flows that runs the full pipeline, yielding step results. Automatically skips register when resuming with a knownblobObjectId.executeRegister()/executeCertify(): Convenience methods on flows that handle signing internally and return typed step results.Smart sliver deduplication:
writeEncodedBlobToNode()now checks metadata and sliver status before uploading, matching the Rust SDK's three-gate optimization (metadata status → size heuristic → per-sliver status). Previously stored slivers are skipped to avoid re-sending large data on resume.New storage node methods:
getSliverStatus()andgetMetadataStatus()for lightweight existence checks (endpoints exist in Rust API but were not in TS SDK).New client methods:
getStorageConfirmations()to re-fetch confirmations from all nodes (for post-upload resume),getBlobObject()to load a blob object by ID.onStep/resumeonwriteBlob()andwriteFiles(): Checkpoint callback and resume support on the all-in-one methods. Both now use the flow internally (single implementation path).computeBlobMetadata()accepts optionalnonce: Allows replaying a saved nonce for deterministic resume on the upload relay path.Test plan
run()iterator yields all 4 steps correctlyexecuteRegister/executeCertifyreturn correct step typesregisteredcheckpoint correctly skips encode+register, only yields uploaded+certifiedwriteBlob()withonStepcallback fires at each steppackages/walrus/examples/resumable/AI Assistance Notice
🤖 Generated with Claude Code