The public faucet appears to become unresponsive to mint requests.
Symptoms
While under load, the faucet
- webpage loads correctly but
- returns 504 after a while when requesting tokens
Logs indicate that proving commences, but no note is ever produced. Its unclear what is going wrong as there are no error logs being produced.
Some potential candidates
- proving is local so instance is overloaded
- errors aren't being logged
- state is desyncing from the chain state
One potential desync way is that the tx reaches the node, but is dropped, or the faucet request is canceled before the faucet state is updated.
We can't really tell without improved logs ala #811.
A potential quickfix is to avoid local state entirely. However this restricts the throughput dramatically.
Logs
Apr 29 16:02:30 ip-32-0-133-125 miden-faucet[3242459]: 2025-04-29T16:02:30.298279Z INFO miden-faucet: bin/faucet/src/handlers.rs:85: Proving and submitting transaction.
Apr 29 16:02:30 ip-32-0-133-125 miden-faucet[3242459]: 2025-04-29T16:02:30.299412Z INFO miden-faucet: bin/faucet/src/handlers.rs:49: Received a request, account_id: 0xcaefc58cafbbdf100000262e2218d1, is_private_note: false, asset_amount: 100
Apr 29 16:02:30 ip-32-0-133-125 miden-faucet[3242459]: 2025-04-29T16:02:30.300626Z INFO prove_program: miden_prover: /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/miden-prover-0.13.0/src/lib.rs:57: new
Apr 29 16:02:30 ip-32-0-133-125 miden-faucet[3242459]: 2025-04-29T16:02:30.300677Z INFO prove_program:execute_program: miden_processor: /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/miden-processor-0.13.0/src/lib.rs:125: new
Apr 29 16:02:30 ip-32-0-133-125 miden-faucet[3242459]: 2025-04-29T16:02:30.473044Z INFO prove_program:execute_program: miden_processor: /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/miden-processor-0.13.0/src/lib.rs:125: close, time.busy: 172ms, time.idle: 25.4µs
Apr 29 16:02:30 ip-32-0-133-125 miden-faucet[3242459]: 2025-04-29T16:02:30.473095Z INFO prove_program: miden_prover: /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/miden-prover-0.13.0/src/lib.rs:71: Generated execution trace of 71 columns and 131072 steps (49% padded) in 172 ms
Apr 29 16:02:30 ip-32-0-133-125 miden-faucet[3242459]: 2025-04-29T16:02:30.473168Z INFO prove_program:build_domain: winter_prover: /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/winter-prover-0.12.2/src/lib.rs:312: new trace_length: 131072, lde_domain_size: 1048576
Apr 29 16:02:30 ip-32-0-133-125 miden-faucet[3242459]: 2025-04-29T16:02:30.477672Z INFO prove_program:build_domain: winter_prover: /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/winter-prover-0.12.2/src/lib.rs:312: close, time.busy: 4.48ms, time.idle: 25.9µs trace_length: 131072, lde_domain_size: 1048576
Apr 29 16:02:30 ip-32-0-133-125 miden-faucet[3242459]: 2025-04-29T16:02:30.477727Z INFO prove_program:commit_to_main_trace_segment: winter_prover: /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/winter-prover-0.12.2/src/lib.rs:497: new
Apr 29 16:02:30 ip-32-0-133-125 miden-faucet[3242459]: 2025-04-29T16:02:30.477757Z INFO prove_program:commit_to_main_trace_segment:extend_execution_trace: winter_prover::trace::trace_lde::default: /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/winter-prover-0.12.2/src/trace/trace_lde/default/mod.rs:258: new num_cols: 71, blowup: 8
The public faucet appears to become unresponsive to mint requests.
Symptoms
While under load, the faucet
Logs indicate that proving commences, but no note is ever produced. Its unclear what is going wrong as there are no error logs being produced.
Some potential candidates
One potential desync way is that the tx reaches the node, but is dropped, or the faucet request is canceled before the faucet state is updated.
We can't really tell without improved logs ala #811.
A potential quickfix is to avoid local state entirely. However this restricts the throughput dramatically.
Logs