Deferred Block Proving

[sergerad]

Deferred Block Proving

1. Background and Constraints

Block proving is expected to take approximately 30 seconds on average per block, while the network target block time is 5 seconds. Therefore, block production MUST NOT synchronously depend on block proving.

The system MUST satisfy the following constraints:

1. Block production proceeds at a fixed cadence independent of proving latency.
2. Committed blocks are immediately visible to the network and RPC clients.
3. Proving is performed asynchronously and may lag behind the committed chain tip.
4. Finality is monotonic and derives solely from proven block prefixes.

These constraints necessitate a separation between block commitment and block proving.

2. High-Level Design Overview

The system introduces asynchronous block proving as a first-class component of the block lifecycle.

Blocks progress through a series of well-defined states. State transitions prior to commitment are synchronous and atomic with respect to chain state. All proving-related transitions are asynchronous and retryable.

Committed blocks represent the canonical chain state, while finalized blocks represent the canonical and proven chain prefix.

sequenceDiagram
    autonumber

    participant BP as Block Producer
    participant V as Validator
    participant ST as Store
    participant PS as Proof Scheduler
    participant RP as Remote Prover

    Note over BP,ST: Block creation phase (synchronous)

    BP->>V: Proposed block
    V->>V: Validate block
    V-->>BP: Signed block

    BP->>ST: Submit signed block
    ST->>ST: Commit block to chain state
    Note over ST: Block committed <br> Visible via RPC

    Note over ST,RP: Block proving phase (asynchronous)

    PS->>ST: Query committed, unproven blocks
    ST-->>PS: Return next committed block

    PS->>RP: Dispatch proving job (signed block)
    RP->>RP: Generate proof

    alt Proof succeeds
        RP-->>PS: Proof
        PS->>ST: Submit proof
        ST->>ST: Mark block as Proven
    else Proof fails
        RP-->>PS: Error
        PS->>PS: Retry with backoff
    end

    Note over ST: Finalization check

    ST->>ST: Check all ancestors proven
    ST->>ST: Mark block as Finalized
    Note over ST: Block finalized <br> Visible via RPC

Loading

3. Block Lifecycle and States

3.1 Block States

The block lifecycle consists of the following ordered states:

Proposed → Signed → Committed → Proven → Finalized

3.2 State Definitions

State	Description	Guarantees	Available via RPC
Proposed	Candidate block produced locally	None	No
Signed	Block signed by a Validator	Authenticity only	No
Committed	Block applied to canonical chain state	Deterministic state transition	Yes
Proven	Block state transition proven	Cryptographic validity	No
Finalized	Proven block with all ancestors proven	Irreversible	Yes

3.3 Invariants

The system MUST uphold the following invariants:

1. The committed chain is always contiguous and strictly ordered by block number.
2. The finalized chain is always a prefix of the committed chain.
3. A block MUST NOT be finalized unless all preceding blocks are proven.
4. Block states MUST NOT regress except via explicit manual intervention (disaster recovery).

4. Block Commitment Flow (Synchronous)

4.1 Commitment Pipeline

Blocks are proposed by the Block Producer, signed by a Validator, and committed by the Store. This flow is synchronous and linear.

4.2 Semantics

The commitment flow is synchronous and atomic.

Upon commitment:

• The block is appended to the canonical chain.
• Chain state is updated immediately.
• All state-dependent RPC endpoints reflect the new block.

Commitment MUST NOT depend on proving.

Once committed, a block is considered part of the canonical chain regardless of its proving status.

5. Asynchronous Block Proving

5.1 Overview

Block proving is performed asynchronously by one or more Remote Provers. Proving is pull-based, retryable, and independent of block production.

5.2 Proving Scheduler

5.2.1 Scheduler Responsibilities

A Proving Scheduler is responsible for:

1. Selecting committed but unproven blocks.
2. Dispatching proving jobs to Remote Provers.
3. Tracking proving progress and failures.
4. Persisting successful proofs.

5.2.2 Scheduler Placement

The scheduler MAY be implemented as an internal task within the Store or as a standalone service consuming Store RPCs.

This specification treats the scheduler as a logical component independent of deployment topology.

5.2.3 Block Selection Rules

The scheduler MUST:

• Only schedule blocks in the Committed state.
• Ensure each block is proven at most once concurrently.
• Prefer lower block numbers to minimize finalized lag.

Blocks MAY be scheduled out of order, but finalization semantics remain strictly sequential.

5.2.4 Parallelism

Multiple blocks MAY be proven in parallel.

Proving MUST NOT assume sequential dependency between blocks, and proof verification MUST be independent per block.

5.2.5 Retry Semantics

Failed proving attempts are retried with backoff.

Transient errors MUST NOT mark a block as unprovable. Permanent failures are escalated according to the failure-handling procedures.

5.3 Proof Persistence

Upon successful proof verification, the Store persists the proof and transitions the block to the Proven state.

6. Block Finalization

6.1 Finalization Rule

A block N is considered finalized if and only if:

1. Block N is in the Proven state; and
2. All blocks with numbers less than N are also in the Proven state.

Finalization is a derived property and does not require an explicit action.

6.2 Finalized Chain Tip

The finalized chain tip is defined as the highest block number such that all blocks up to and including that number are proven.

7. RPC Semantics and Finality

7.1 Finality Parameter

RPC requests that reference block numbers MUST specify a finality requirement:

• committed
• finalized

The finality parameter MUST NOT default implicitly.

7.2 RPC Behavior

If the requested block does not satisfy the requested finality, the RPC MUST return a deterministic error (for example, PENDING_FINALITY).

RPC responses MUST NOT mix data across different finality levels.

7.3 Affected Store RPC endpoints

The following RPC endpoints are implicated by finality semantics:

• GetAccount
• GetBlockByNumber
• GetBlockHeaderByNumber
• SyncNullifiers
• SyncNotes
• SyncState
• SyncAccountVault
• SyncAccountStorageMaps
• SyncTransactions

All RPC endpoints upstream of these are also impacted (the service Api endpoints in rpc.proto).

7.4 Store RPC for Block Producer and NTX Builder

The Block Producer and NTX Builder require only committed data and MUST NOT depend on proven or finalized blocks.

8. Observability and Operational Expectations

The system SHOULD expose metrics for:

• Committed versus finalized chain tip
• Proving backlog size
• Proof latency distributions
• Persistent proving failures

Alerting SHOULD trigger when the finalized tip lags the committed tip beyond a configured threshold.

9. Failure Handling

9.1 Irrecoverable Errors

In extremely rare cases, a block MAY be unprovable due to a genuine fault in its state transition.

In such cases:

• Block production MUST be halted.
• Manual intervention is REQUIRED.

Remediation MAY involve reverting the chain to a prior committed height or canonicalizing the faulty transition via governance or override mechanisms.

9.1.1 Automatic Detection and Alerting

Such irrecoverable errors MUST be automatically detected by the system and raised as an operational alert.

9.1.2 Manual Rollback

The system MUST allow for operators to perform a rollback of chain state in such instances when it is deemed that an error is irrecoverable.

9.2 Recoverable Errors

All other proving failures are considered recoverable and MUST be handled automatically after a patch or configuration change.

Recoverable errors MUST NOT permanently stall block production.

10. Open Questions

1. Final scheduler placement: embedded Store task versus standalone service.

11. Plan

11.1 Proving Control Plane

1. Implement block proof scheduler.
   • a. Define scheduler interface (inputs, outputs, persistence model).
   • b. Implement block selection heuristics:
     • committed but unproven blocks only
     • lowest block number first
     • at most one in-flight proof per block
   • c. Implement parallel proof job dispatch:
     • configurable concurrency limits
     • backpressure when prover capacity is saturated
   • d. Implement retry and backoff semantics for transient failures.
   • e. Implement permanent failure escalation path (unprovable block signal).

Deliverable: Scheduler can continuously drain the committed → proven backlog without affecting block production.

11.2 Proof Persistence

2. Integrate proofs into the Store.
   • a. Persist proof metadata as required.
   • b. Enforce idempotency for duplicate proof submissions.
   • c. Prevent state regression or double-proving.

Deliverable: Proven blocks are durably recorded and safe against replay or scheduler restarts.

11.3 Finalization Logic

3. Implement finalized chain tip logic.
   • a. Compute finalized height as the maximal contiguous prefix of proven blocks.
   • b. Ensure finalization is monotonic and deterministic.
   • c. Ensure finalization derivation is independent of scheduler ordering.

Deliverable: Finalized chain tip is correct under reordering, retries, and partial proving.

11.4 RPC Finality Semantics

4. Extend Store and external RPCs with explicit finality semantics.
   • a. Add a finality parameter (committed | finalized) to all block-indexed RPCs.
   • b. Enforce deterministic errors when requested finality is not satisfied.
   • c. Ensure no cross-finality data leakage in RPC responses.
   • d. Ensure Block Producer and NTX Builder call paths remain committed-only.

Deliverable: RPC consumers can safely choose between low-latency and fully finalized views.

11.5 Operational Visibility and Safety

5. Implement observability and alerting.
   • a. Metrics:
     • committed vs finalized chain tip
     • proving backlog size
     • in-flight proofs
     • proof latency and failure rates
   • b. Alerts when finalized lag exceeds configured thresholds.

Deliverable: Operators can reason about proving health based on metrics and alerts.

11.6 Failure and Recovery Procedures

6. Implement state rollback and recovery mechanisms.
   • a. Mechanism to halt block production and finalization without bringing node down.
   • b. Tool to execute rollback.
   • c. Operational runbook(s).

Deliverable: Operational readiness for irrecoverable errors.

[bobbinth]

Thank you for writing this up in such detail! A few thoughts/comments:

Proposed → Signed → Committed → Proven → Finalized

Overall, this is fine, but I wouldn't use the word "finalized" here. Finalized would mean that we have posted the proofs to L1 (and in the future, this may change to rely on our own consensus). From the database perspective, we'll probably have blocks in 3 states:

• committed - stored in the DB without the proof.
• proven - stored in the DB with the proof.
• finalized - data posted to and finalized on L1 (for now, unused).

The scheduler MAY be implemented as an internal task within the Store or as a standalone service consuming Store RPCs.

I would probably implement the scheduler as in internal task (at least for now).

Upon successful proof verification, the Store persists the proof and transitions the block to the Proven state.

I would persist proofs only once all ancestor blocks have valid proofs as well. For example, let's say we are proving 3 blocks in parallel - 101, 102, and 103. If the proof for 102 gets done first, we keep it marked as proven in memory and only persist the proof once we also have the proof for block 101. I think this should simplify the overall design as we would never have blocks with un-proven ancestors in the DB.

Separate question: where are we thinking of persisting the proofs? These proofs would probably be ~200KB. We could save them into the database or on disk.

The following RPC endpoints are implicated by finality semantics:

• GetAccount
• GetBlockByNumber
• GetBlockHeaderByNumber
• SyncNullifiers
• SyncNotes
• SyncState
• SyncAccountVault
• SyncAccountStorageMaps
• SyncTransactions

I would love to avoid modifying all these endpoints. We may be able to get away with modifying only the SyncChainMmr endpoint (which replaced the SyncState). The way it could work would be as follows:

• The client would first call SyncChainMmr and select whether it wants to sync to committed/proven/finalized chain tip.
• Then, for other sync endpoints, the client would use the block_to field to specify which block they'd like to sync to.

Remediation MAY involve reverting the chain to a prior committed height or canonicalizing the faulty transition via governance or override mechanisms.

I would split out rolling back blocks into a separate issue and tackle it separately (probably in the next milestone).

---

Overall, I think we have the following tasks/questions:

1. Figure out where we want to store the proofs.
2. Update block_header table to track block state/status info.
3. Update SyncChainMmr endpoint to specify the level of "finality" for the chain tip we want to get.
4. Implement the scheduler to dispatch proof requests, collect responses, and persist block proofs (once all ancestor blocks also have been proven).
5. Detect unrecoverable failures in the scheduler and shut down the node.

Also, for bikeshedding: I don't love the "scheduler" name - but don't have anything better at the moment.

[Mirko-von-Leipzig]

Final scheduler placement: embedded Store task versus standalone service.

I think this 100% should be a basic task within the store. The simpler the better. The way I'd imagined it in my head:

• standalone task which gets notified of new committed blocks via a channel from apply_block
• each "proof" is a future (future would include retries etc, and an overall timeout)
• FIFO order is maintained using something like FuturesOrdered
• above ordered queue has limited capacity to ensure there is backpressure
• fatal error ends the task which kills the store
  • store will restart automatically and then we need to preload these as part of initialization
  • if we want fatal to be fatal until manual intervention then we will need to mark the database as dirty somehow, though I'm unsure that we really want this.

---

Figure out where we want to store the proofs.

I guess a question is -- who needs access to these, aka what is the read utilisation like.

From sqlite's own benchmarks on BLOB size:

We deduce the following rules of thumb from the matrix above:

• A database page size of 8192 or 16384 gives the best performance for large BLOB I/O.
• For BLOBs smaller than 100KB, reads are faster when the BLOBs are stored directly in the database file. For BLOBs larger than 100KB, reads from a separate file are faster.

Of course, your mileage may vary depending on hardware, filesystem, and operating system. Double-check these figures on target hardware before committing to a particular design.

Something additional to note is that keeping the proofs in sqlite means we may clash with apply_block writes.