Developer mode for the node

[Mirko-von-Leipzig]

Block and batch production is currently automatic, every M seconds a batch build is triggered, and every N a block.

This makes it difficult to get the chain into a specific state since batches and blocks are somewhat random. It would be useful for integration tests, demonstrations and test tooling for builders if this was more controllable.

I propose an additional manual block-producer mode. When launched in this mode, the block-producer does not spin up batch and block producer tasks, but instead launches a server to accept commands from.

The first concrete step of this issue should be a API design, but I imagine we could start with a simple /build_batch and /build_block. Maybe we want to allow specifying number of transactions/batches to select, or even which ones to include.

It would also be ideal if the user of this API did not need to know about its specifics, and could instead use CLI to control it. This should be described in the initial design proposal.

---

After some discussion with @sergerad, it sounds like we should consider something a bit more extensive in a similar vein to foundry's anvil for ethereum development.

This could include support for

• local node (essentially what we have now)
• manual mode/control (initial proposal)
• forking existing network at some block height

I'm thinking we could provide a cohesive crate API which would be used by

• the formal node
• new cli akin to anvil (could also be part of the node binary if we choose)

which ensures that the same underlying code is used by both the formal node and the tooling around it.

This could cover use cases like:

• basis for our own integration tests, with tighter control over what gets produced when, instead of fighting the concurrency of normal node operation
• dApp testing
• deterministic bug debugging. One can now provide a reproducible example of a bug or issue or dx concern, e.g. testnet tx fails, so user decides to reproduce by creating a script that
  • forks at the block height
  • submits the tx
  • which fails
  • and can be rerun indefinitely

---

I'd appreciate input on what flows would be nice to have from a dx / user perspective. I don't have enough experience with foundry etc to understand potential issues and desires here.

In my mind the main difference in execution is that the dev-node would always operate sequentially in order to be repeatable.

[partylikeits1983]

In the current form of the miden node, it's quite easy to start a local Miden node and then connect the Rust client to it.

Here's an example to show that its is quite simple. Generate the genesis file and initialize the data and accounts directories:

miden-node store dump-genesis > genesis.toml
mkdir -p data accounts
miden-node bundled bootstrap \
  --data-directory data \
  --accounts-directory accounts \
  --config genesis.toml

Then, start the node:

miden-node bundled start \
  --data-directory data \
  --rpc.url http://0.0.0.0:57123

Voila, the node is running locally :)

This is how you instantiate the Rust client against this local node:

let endpoint = Endpoint::new(
    "http".to_string(),
    "localhost".to_string(),
    Some(57123),
);
let timeout_ms = 10_000;
let rpc_api = Arc::new(TonicRpcClient::new(&endpoint, timeout_ms));

let mut client = ClientBuilder::new()
    .with_rpc(rpc_api)
    .with_filesystem_keystore("./keystore")
    .in_debug_mode(true)
    .build()
    .await?;

Given how easy it is to run the node locally, I don't think there is actually too much left to be done to make the miden-node have a similar functionality as Anvil.

Ideas for “testing mode” features

It would be beneficial to have the following features to in the miden-node's "testing mode":

1. Fast-forward control
   — Ability to advance (monotonically) to a specific block number or timestamp.
2. One-step startup
   — A single command (e.g. miden-node start --testing -a 10 -f 2) that:
   • Spins up the node in testing mode
   • Pre-funds 10 accounts
   • Distributes two different assets across those accounts
3. Concurrent integration tests
   — Allow the Miden client and node to run multithreaded integration tests, instead of only sequentially.

---

However, to make the dev mode for the node work seamlessly in the test environment of miden-app it will also require a bit of coordination with the test harness @bitwalker is working on. Also the test harness should work with the rust client so that when developers are writing unit & integration tests, they don't have to completely rewrite logic for their production application.

[Mirko-von-Leipzig]

Here's an example for to show that its is quite simple. Generate the genesis file and initialize the data and accounts directories:

btw we removed the genesis file entirely for the next release. This simplifies general startup, but maybe its a bad idea because genesis accounts can no longer be customized?

Given how easy it is to run the node locally, I don't think there is actually too much left to be done to make the miden-node have a similar functionality as Anvil.

There isn't much underlying code to change; the main changes would be allowing more manual control of block creation. But we are quite well setup to do this, which is why I thought we may as well :) Question now is just, what exactly is required as an API.

Fast-forward control
— Ability to advance (monotonically) to a specific block number or timestamp.

Doable. Is there value in more fine grained control? i.e. stop manual block/batch creation, add commands to create a block/batch, optionally with specific transaction/batch IDs?

Timestamp.. might be a bit hazardous. Its probably doable, but we would have to consider how the timestamp should progress after its been manually altered, and how to do this without breaking other things. e.g. timestamps in logs would not be similarly altered but would reflect system time.

One-step startup
— A single command (e.g. miden-node start --testing -a 10 -f 2) that:

• Spins up the node in testing mode
• Pre-funds 10 accounts
• Distributes two different assets across those accounts

I don't quite understand this. What assets, and how many amounts? I think in other chains this is more obvious, i.e. on ethereum you add Eth.

I guess I'm unsure it could be this trivial, but I guess one can predefine assets.

Concurrent integration tests
— Allow the Miden client and node to run multithreaded integration tests, instead of only sequentially.

Concurrent as in multiple dev-nodes? Or concurrent as in the client should be concurrent? I'm assuming there isn't really much to do on the node side, but rather on the client/test itself.

[Mirko-von-Leipzig]

However, to make the dev mode for the node work seamlessly in a miden-app it will also require a bit of coordination with the test harness @bitwalker is working on. Also the test harness should work with the rust client so that when developers are writing unit & integration tests, they don't have to completely rewrite logic for their production application.

My initial thoughts here were to only provide access via the CLI i.e. out-of-process. But it would probably make life easier if the node is directly controllable in the code. This would allow specifying account code in rust with components, injecting them into the genesis block etc, all without leaving rust.

[Mirko-von-Leipzig]

Example node integration test

// This is a test to ensure our open-telemetry trace spans are correct for a block build.
//
// More complexity might be required to test negatives aka error reporting, because those
// _should_ only occur when we have a bug. So we may have to corrupt the database or something
// to force such a condition as part of setup. Not dealing with that here though.

// Collects he OTel spans for this test.
let otel_collector = ..;

// Setup test conditions; create chain with enough state to be useful i.e. with a little bit of
// everything. Variety of accounts, notes, nullifiers and txs.
let chain = MockChain::builder()
  .with_default_genesis()
  .push_block(...)
  .push_block(...)
  .build()
  .expect("blocks are valid")'

// Launch the node in manual mode.
let mut store = Store::builder()
  .in_memory()
  .with_otel_collector(otel_collector.clone())
  ,build();

// Inject the chain blocks into the node's store.
for block in chain {
  // This is equivalent to gRPC call to store.apply_block.
  store.apply_block(block).unwrap();
}

// Create the block-producer in manual mode aka no automatic block-production.
// This is to make the test more deterministic.
let mut bp = BlockProducer::builder()
  .manual_mode()
  .with_store(store.grpc_endpoint())
  .with_otel_collector(otel_collector.clone())
  .build()
  .unwrap();

// Create the block we want to test.
// This also requires knowing the batches etc as well.
// I don't believe this exists at the moment.
let txs = chain... // Somehow create them using the chain.
// Insert into mempool.
for tx in txs {
  // Equivalent to the grpc method.
  bp.add_tx(tx).unwrap();
}

// Create the batches.
let (batch_0, batch_1) = txs.split_at(5);
let batch_id_0 = bp.add_batch(batch_0.map(Transaction::id)).unwrap();
let batch_id_1 = bp.add_batch(batch_1.map(Transaction::id)).unwrap();

// Create block (actual test bit)
// Drain all spans until now since they're not part of 
// this test.
otel_collector.flush();

bp.create_block(vec![batch_id_0, batch_id_1]).unwrap();
let spans = otel_collector.flush().as_json();

// Expected output, retrieved as a snapshot.
let expected = insta!(...);

assert_eq!(spans, expected);

[igamigo]

One related feature that would be nice to have is to have the miden-node crate expose some functionality to easily start a bundled node. Right now, if someone wants to bootstrap their own node for testing purposes, you need to start each component individually and this comes with its complexities (we do something like this in the client; not too hard to maintain but still having a simpler bundled way with good defaults would be better). It could be a builder-like struct that could even take a MockChain or something similar for bootstrapping the state as suggested above.

[Mirko-von-Leipzig]

I do like this idea; we just need to take care with order of operations for now because the component start up order is important - and critically can change e.g. ntb requires store sync before block production begins now.

I think in general we would need to properly decouple the components so we can have a simpler API here. That said, we can definitely do something already, I just think it will be easier if one doesn't have a specific "boot sequence" to also worry about.

[Mirko-von-Leipzig]

But this would be really cool for integration tests both internally for the node, and externally like you want for the client.