continuation optimization

[dajuguan]

Presently, when zkwasm's instructions exceed 2 billion (as observed in zkGo), the generated trace table becomes too large to fit into memory. Moreover, the generation of the witness table consumes a considerable amount of time, taking, for instance, up to 7 hours for 2 billion instructions. This pull request aims to optimize several aspects:

1. Implementing the capability to dump the trace table and reload it to reconstruct the circuit accurately. A specific test case test_rlp_from_file will be provided to ensure the outcome aligns with test_rlp_slice, as in 1rd commit.

2. Introducing a tracer callback mechanism to dump tables periodically per compute_slice_capability function's output, as in the 2rd commit. Note that there is also a related pr in wasmi repo. Notably，wasm's maximum memory has been hard-coded to 64MB via LINEAR_MEMORY_MAX_PAGES , otherwise, the current implementation will incur OOM due to push_init_memory pushes all the memory into imtable.

3. Adding support for binary files as private inputs for scenarios involving large input sizes. The new arg is --private <filename>:file, as in the 3rd commit.

4. Further optimizing witness generation time through strategies such as caching, flexbuffer utilization, and pooling, as in this commit

5. [WIF] reconstruct code

[xgaozoyoe]

small adjustments are needed.

[xgaozoyoe]

Implementing the capability to dump the trace table and reload it to reconstruct the circuit accurately. A specific test case test_rlp_from_file will be provided to ensure the outcome aligns with test_rlp_slice.

I can not figure out how the output of test_rlp_slice and test_rlp_from_file are compared.

Introducing a tracer callback mechanism to dump tables periodically, say, after every 1,000,000 instructions.
Would you please say a bit more about how to use the tracer callback? Is this included in this pr?

Adding support for binary files as private inputs for scenarios involving large input sizes
Is this included in this pr?

Further optimizing witness generation time through strategies such as caching, flexbuffer utilization, and pooling

Is this included in this pr?

If this is a WIP PR, please mark it as WIP.

[dajuguan]

Presently, when zkwasm's instructions exceed 2 billion (as observed in zkGo), the generated trace table becomes too large to fit into memory. Moreover, the generation of the witness table consumes a considerable amount of time, taking, for instance, up to 7 hours for 2 billion instructions. This pull request aims to optimize several aspects:

1. Implementing the capability to dump the trace table and reload it to reconstruct the circuit accurately. A specific test case test_rlp_from_file will be provided to ensure the outcome aligns with test_rlp_slice, as in the current commit.
2. Introducing a tracer callback mechanism to dump tables periodically, say, after every 1,000,000 instructions. This approach resembles the methodology outlined in this commit.
3. Adding support for binary files as private inputs for scenarios involving large input sizes, mirroring the implementation in this commit.
4. Further optimizing witness generation time through strategies such as caching, flexbuffer utilization, and pooling, similar to the enhancements introduced in this commit

Implementing the capability to dump the trace table and reload it to reconstruct the circuit accurately. A specific test case test_rlp_from_file will be provided to ensure the outcome aligns with test_rlp_slice.

I can not figure out how the output of test_rlp_slice and test_rlp_from_file are compared.

Introducing a tracer callback mechanism to dump tables periodically, say, after every 1,000,000 instructions.
Would you please say a bit more about how to use the tracer callback? Is this included in this pr?

Adding support for binary files as private inputs for scenarios involving large input sizes
Is this included in this pr?

Further optimizing witness generation time through strategies such as caching, flexbuffer utilization, and pooling

Is this included in this pr?

If this is a WIP PR, please mark it as WIP.

Marked the as WIPs.

[dajuguan]

Implementing the capability to dump the trace table and reload it to reconstruct the circuit accurately. A specific test case test_rlp_from_file will be provided to ensure the outcome aligns with test_rlp_slice.

I can not figure out how the output of test_rlp_slice and test_rlp_from_file are compared.

Introducing a tracer callback mechanism to dump tables periodically, say, after every 1,000,000 instructions.
Would you please say a bit more about how to use the tracer callback? Is this included in this pr?

Adding support for binary files as private inputs for scenarios involving large input sizes
Is this included in this pr?

Further optimizing witness generation time through strategies such as caching, flexbuffer utilization, and pooling

Is this included in this pr?

If this is a WIP PR, please mark it as WIP.

Regarding comparing test_rlp_from_file with test_rlp_slice, test_rlp_from_file operates by extracting tables from Slice, reconstructing the circuit using the dumped file, and subsequently validating each slice's integrity via its respective mock_test, mirroring the process of test_rlp_slice.

Others are marked as WIP.

[junyu0312]

replace context_out with context_output and remove context_out variable?

[junyu0312]

If it’s necessary, could you move the function to specs/src/mtable.rs? I’m curious why it was moved to the specs crate.