Faucet desyncs

[Mirko-von-Leipzig]

The public faucet appears to become unresponsive to mint requests.

Symptoms

While under load, the faucet

• webpage loads correctly but
• returns 504 after a while when requesting tokens

Logs indicate that proving commences, but no note is ever produced. Its unclear what is going wrong as there are no error logs being produced.

Some potential candidates

• proving is local so instance is overloaded
• errors aren't being logged
• state is desyncing from the chain state

One potential desync way is that the tx reaches the node, but is dropped, or the faucet request is canceled before the faucet state is updated.

We can't really tell without improved logs ala #811.

A potential quickfix is to avoid local state entirely. However this restricts the throughput dramatically.

Logs

Apr 29 16:02:30 ip-32-0-133-125 miden-faucet[3242459]: 2025-04-29T16:02:30.298279Z  INFO miden-faucet: bin/faucet/src/handlers.rs:85: Proving and submitting transaction.
Apr 29 16:02:30 ip-32-0-133-125 miden-faucet[3242459]: 2025-04-29T16:02:30.299412Z  INFO miden-faucet: bin/faucet/src/handlers.rs:49: Received a request, account_id: 0xcaefc58cafbbdf100000262e2218d1, is_private_note: false, asset_amount: 100
Apr 29 16:02:30 ip-32-0-133-125 miden-faucet[3242459]: 2025-04-29T16:02:30.300626Z  INFO prove_program: miden_prover: /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/miden-prover-0.13.0/src/lib.rs:57: new
Apr 29 16:02:30 ip-32-0-133-125 miden-faucet[3242459]: 2025-04-29T16:02:30.300677Z  INFO prove_program:execute_program: miden_processor: /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/miden-processor-0.13.0/src/lib.rs:125: new
Apr 29 16:02:30 ip-32-0-133-125 miden-faucet[3242459]: 2025-04-29T16:02:30.473044Z  INFO prove_program:execute_program: miden_processor: /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/miden-processor-0.13.0/src/lib.rs:125: close, time.busy: 172ms, time.idle: 25.4µs
Apr 29 16:02:30 ip-32-0-133-125 miden-faucet[3242459]: 2025-04-29T16:02:30.473095Z  INFO prove_program: miden_prover: /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/miden-prover-0.13.0/src/lib.rs:71: Generated execution trace of 71 columns and 131072 steps (49% padded) in 172 ms
Apr 29 16:02:30 ip-32-0-133-125 miden-faucet[3242459]: 2025-04-29T16:02:30.473168Z  INFO prove_program:build_domain: winter_prover: /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/winter-prover-0.12.2/src/lib.rs:312: new trace_length: 131072, lde_domain_size: 1048576
Apr 29 16:02:30 ip-32-0-133-125 miden-faucet[3242459]: 2025-04-29T16:02:30.477672Z  INFO prove_program:build_domain: winter_prover: /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/winter-prover-0.12.2/src/lib.rs:312: close, time.busy: 4.48ms, time.idle: 25.9µs trace_length: 131072, lde_domain_size: 1048576
Apr 29 16:02:30 ip-32-0-133-125 miden-faucet[3242459]: 2025-04-29T16:02:30.477727Z  INFO prove_program:commit_to_main_trace_segment: winter_prover: /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/winter-prover-0.12.2/src/lib.rs:497: new
Apr 29 16:02:30 ip-32-0-133-125 miden-faucet[3242459]: 2025-04-29T16:02:30.477757Z  INFO prove_program:commit_to_main_trace_segment:extend_execution_trace: winter_prover::trace::trace_lde::default: /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/winter-prover-0.12.2/src/trace/trace_lde/default/mod.rs:258: new num_cols: 71, blowup: 8

[igamigo]

Likely not desyncing from the chain because there are transactions going through and updating the faucet account (block committed ~1 minute ago). Notes seem to be created as well (example).
Probably the request queue is backed up and requests time out before getting a hold of the executor and prover.

[Mirko-von-Leipzig]

Likely not desyncing from the chain because there are transactions going through and updating the faucet account (block committed ~1 minute ago). Notes seem to be created as well (example). Probably the request queue is backed up and requests time out before getting a hold of the executor and prover.

Its because we're force restarting the faucet every 5 minutes :D

Without this it just stalls forever.

The execution and proving is also done locally afaik :/ And network is limited to 8 txs per block so txs/s are max 1.3s. Lots of potential issues.

[igamigo]

I see. I was thinking since the sync only happens at startup and transactions are going through, it must've not desynced and so the bottleneck would just be the local proving aspect.