Skip to content

feat(evm): add witness oracle metrics - #983

Draft
0xOsiris wants to merge 1 commit into
mainfrom
feat/witness-oracle-consumer
Draft

feat(evm): add witness oracle metrics#983
0xOsiris wants to merge 1 commit into
mainfrom
feat/witness-oracle-consumer

Conversation

@0xOsiris

@0xOsiris 0xOsiris commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Adds the missing observability for the live pre-image witness oracle (--witness.collect). Phase 5 (host integration) is deliberately not built — see the verdict below.

What this adds

New world_chain.witness metric scope (crates/evm/src/metrics.rs):

metric why
captured / dropped WorldChainBlockExecutor::finish uses try_send; on a full channel it drops the witness and continues. A single drop makes every range spanning that block permanently unservable, and today that is invisible.
assembly_failed Same permanent-hole failure mode, from the collector side.
inserted / evicted / cache_len / cache_oldest_block / cache_newest_block Cache bounds are the first thing you need during triage: "is the range I want even in the window?"
witness_bytes (histogram) / cache_bytes The real per-block witness size on World Chain is not documented anywhere. This measures it in production so the memory cost of --witness.depth is a number instead of a guess.
range_hit / range_miss / range_missing_blocks Hit rate of debug_collectRangeWitness, and how far off a miss was.

WitnessCache now tracks retained bytes exactly across replace and eviction (new unit test covers it). All counters are inert when --witness.collect is off.

Verdict on Phase 5: do not build it yet

op-reth already ships --proofs-history, and it subsumes the witness cache for the Kona host path:

  • rust/op-reth/crates/node/src/proof_history.rs:94-101 installs RPC overrides for eth_getProof and debug_executePayloaddebug_executePayload is the exact high-level hint the witness cache targets (kona/bin/host/src/single/handler.rs:394-421), and eth_getProof is what the current incident is about (handler.rs:296-341, :225-241).
  • It is disk-backed MDBX with a 1,296,000-block (~30 day) default window, plus op-proofs backfill/prune/snapshot subcommands — versus 1024 blocks (~34 min) of RAM, lost on every restart, with no backfill.
  • It needs zero Kona-host changes: build_range_input keeps talking standard RPC.

The witness cache has one genuine advantage — zero re-execution on the serving side (op-reth still re-executes in DebugApiExt::execute_payload, crates/rpc/src/debug.rs:194-255, it just makes the state reads fast). That advantage is real but does not justify a second, weaker mechanism before the first one is even switched on.

World Chain parses --proofs-history today and silently ignores it. WorldChainArgs.rollup is op-reth RollupArgs (crates/cli/src/cli.rs:8,167), which carries proofs_history (op-reth/crates/node/src/args.rs:196), so the flags show up in specs/cli/reference.md:79-106. World Chain only ever reads sequencer, disable_txpool_gossip, discovery_v4 and compute_pending_block from it — the exex and the RPC overrides are never installed. An operator who sets --proofs-history gets a node that looks configured and is not. Fixing that is the higher-priority change and it subsumes most of Phase 5.

Fix concerns 2 (single-drop poisoning, no repair path) and 3 (metrics — this PR) before revisiting the witness cache at all.

🤖 Generated with Claude Code


Note

Low Risk
Observability-only changes with no altered witness serving logic; cache byte accounting is additive and covered by a new test.

Overview
Adds world_chain.witness Prometheus metrics for the live pre-image witness path (--witness.collect), so drops, cache bounds, memory use, and range lookup success are visible in production.

WitnessMetrics (lazy singleton) records capture (captured / dropped), collector assembly failures (assembly_failed), cache lifecycle (inserted, evicted, witness_bytes, cache_bytes, cache_len, oldest/newest block gauges), and WitnessCache::range outcomes (range_hit, range_miss, range_missing_blocks). WorldChainBlockExecutor::finish and the witness collector increment the failure counters; the cache updates gauges on insert and range.

WitnessCache now keeps an exact running byte total per entry (retained_bytes over state/codes/keys/headers), including on replace and eviction, with a unit test for that accounting. metrics / metrics-derive are added to world-chain-evm.

Reviewed by Cursor Bugbot for commit 55005fe. Configure here.

@github-actions

github-actions Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results

Base and PR measured on the same runner in the same workflow run.

Benchmark Base PR Change
flashblock_validation_launch_flashblock_sequence_eth_transfers/stream/4fb_x_125tx 28.5±0.06ms 29.6±0.20ms 🟡 +4.00%
flashblock_validation_launch_flashblock_sequence_eth_transfers/stream/4fb_x_250tx 58.4±0.21ms 58.8±0.44ms 🟡 +1.00%
flashblock_validation_launch_flashblock_sequence_eth_transfers/stream/4fb_x_50tx 11.8±0.03ms 12.1±0.09ms 🟡 +2.00%
flashblock_validation_launch_flashblock_sequence_eth_transfers_with_bal/stream/4fb_x_125tx 34.8±0.23ms 35.2±0.25ms 🟡 +1.00%
flashblock_validation_launch_flashblock_sequence_eth_transfers_with_bal/stream/4fb_x_250tx 74.5±0.48ms 75.5±0.53ms 🟡 +1.00%
flashblock_validation_launch_flashblock_sequence_eth_transfers_with_bal/stream/4fb_x_50tx 14.2±0.09ms 14.8±0.08ms 🟡 +5.00%
flashblock_validation_launch_flashblock_sequence_fib/stream/4fb_x_125tx 27.1±0.06ms 27.0±0.11ms 🟢 +0.00%
flashblock_validation_launch_flashblock_sequence_fib/stream/4fb_x_250tx 52.9±0.12ms 53.1±0.11ms 🟢 +0.00%
flashblock_validation_launch_flashblock_sequence_fib/stream/4fb_x_50tx 11.4±0.04ms 11.2±0.10ms 🟢 -0.99%
flashblock_validation_launch_flashblock_sequence_fib_with_bal/stream/4fb_x_125tx 30.0±0.23ms 29.8±0.44ms 🟢 -0.99%
flashblock_validation_launch_flashblock_sequence_fib_with_bal/stream/4fb_x_250tx 55.5±0.17ms 57.0±1.02ms 🟡 +3.00%
flashblock_validation_launch_flashblock_sequence_fib_with_bal/stream/4fb_x_50tx 13.4±0.12ms 13.4±0.12ms 🟢 +0.00%
flashblock_validation_launch_flashblock_sequence_world_id_like_bn254/stream/4fb_x_10tx 136.0±0.07ms 136.5±0.15ms 🟢 +0.00%
flashblock_validation_launch_flashblock_sequence_world_id_like_bn254/stream/4fb_x_12tx 163.0±0.11ms 163.3±0.22ms 🟢 +0.00%
flashblock_validation_launch_flashblock_sequence_world_id_like_bn254/stream/4fb_x_5tx 68.5±0.22ms 68.8±0.10ms 🟢 +0.00%
flashblock_validation_launch_flashblock_sequence_world_id_like_bn254_with_bal/stream/4fb_x_10tx 22.7±1.54ms 22.7±1.34ms 🟢 +0.00%
flashblock_validation_launch_flashblock_sequence_world_id_like_bn254_with_bal/stream/4fb_x_12tx 21.3±1.07ms 24.0±2.06ms 🔴 +13.00%
flashblock_validation_launch_flashblock_sequence_world_id_like_bn254_with_bal/stream/4fb_x_5tx 18.9±0.52ms 19.5±0.40ms 🟡 +3.00%
flashblock_validation_process_flashblock_eth_transfers/txs/1000 49.6±0.22ms 49.6±0.14ms 🟢 +0.00%
flashblock_validation_process_flashblock_eth_transfers/txs/50 2.8±0.42ms 2.7±0.02ms 🟢 -4.76%
flashblock_validation_process_flashblock_eth_transfers/txs/500 25.2±0.88ms 24.9±0.07ms 🟢 -0.99%
flashblock_validation_process_flashblock_eth_transfers_with_bal/txs/1000 59.5±14.13ms 51.2±0.28ms 🟢 -13.79%
flashblock_validation_process_flashblock_eth_transfers_with_bal/txs/50 3.5±0.53ms 3.2±0.05ms 🟢 -9.09%
flashblock_validation_process_flashblock_eth_transfers_with_bal/txs/500 26.2±1.01ms 25.9±0.33ms 🟢 -0.99%
flashblock_validation_process_flashblock_fib/txs/1000 48.5±0.10ms 48.4±0.24ms 🟢 +0.00%
flashblock_validation_process_flashblock_fib/txs/50 2.7±0.18ms 2.6±0.03ms 🟢 -1.96%
flashblock_validation_process_flashblock_fib/txs/500 26.7±4.71ms 24.4±0.05ms 🟢 -9.09%
flashblock_validation_process_flashblock_fib_with_bal/txs/1000 50.7±2.79ms 49.3±0.22ms 🟢 -2.91%
flashblock_validation_process_flashblock_fib_with_bal/txs/50 3.2±0.37ms 3.2±0.04ms 🟢 -0.99%
flashblock_validation_process_flashblock_fib_with_bal/txs/500 25.9±1.96ms 25.5±0.99ms 🟢 -1.96%
flashblock_validation_process_flashblock_world_id_like_bn254/txs/10 36.3±4.08ms 34.1±0.16ms 🟢 -5.66%
flashblock_validation_process_flashblock_world_id_like_bn254/txs/25 84.7±1.17ms 84.6±0.12ms 🟢 +0.00%
flashblock_validation_process_flashblock_world_id_like_bn254/txs/50 172.4±5.18ms 169.0±0.20ms 🟢 -1.96%
flashblock_validation_process_flashblock_world_id_like_bn254_with_bal/txs/10 6.7±0.17ms 5.6±0.31ms 🟢 -16.67%
flashblock_validation_process_flashblock_world_id_like_bn254_with_bal/txs/25 7.9±0.11ms 8.0±0.04ms 🟡 +1.00%
flashblock_validation_process_flashblock_world_id_like_bn254_with_bal/txs/50 13.0±1.19ms 10.8±0.18ms 🟢 -17.36%

🔴 regression > 10%  ·  🟡 slower  ·  🟢 faster. A regression > 10% fails the check.

Raw critcmp output
group                                                                                              base                                   pr
-----                                                                                              ----                                   --
flashblock_validation_launch_flashblock_sequence_eth_transfers/stream/4fb_x_125tx                  1.00     28.5±0.06ms        ? ?/sec    1.04     29.6±0.20ms        ? ?/sec
flashblock_validation_launch_flashblock_sequence_eth_transfers/stream/4fb_x_250tx                  1.00     58.4±0.21ms        ? ?/sec    1.01     58.8±0.44ms        ? ?/sec
flashblock_validation_launch_flashblock_sequence_eth_transfers/stream/4fb_x_50tx                   1.00     11.8±0.03ms        ? ?/sec    1.02     12.1±0.09ms        ? ?/sec
flashblock_validation_launch_flashblock_sequence_eth_transfers_with_bal/stream/4fb_x_125tx         1.00     34.8±0.23ms        ? ?/sec    1.01     35.2±0.25ms        ? ?/sec
flashblock_validation_launch_flashblock_sequence_eth_transfers_with_bal/stream/4fb_x_250tx         1.00     74.5±0.48ms        ? ?/sec    1.01     75.5±0.53ms        ? ?/sec
flashblock_validation_launch_flashblock_sequence_eth_transfers_with_bal/stream/4fb_x_50tx          1.00     14.2±0.09ms        ? ?/sec    1.05     14.8±0.08ms        ? ?/sec
flashblock_validation_launch_flashblock_sequence_fib/stream/4fb_x_125tx                            1.00     27.1±0.06ms        ? ?/sec    1.00     27.0±0.11ms        ? ?/sec
flashblock_validation_launch_flashblock_sequence_fib/stream/4fb_x_250tx                            1.00     52.9±0.12ms        ? ?/sec    1.00     53.1±0.11ms        ? ?/sec
flashblock_validation_launch_flashblock_sequence_fib/stream/4fb_x_50tx                             1.01     11.4±0.04ms        ? ?/sec    1.00     11.2±0.10ms        ? ?/sec
flashblock_validation_launch_flashblock_sequence_fib_with_bal/stream/4fb_x_125tx                   1.01     30.0±0.23ms        ? ?/sec    1.00     29.8±0.44ms        ? ?/sec
flashblock_validation_launch_flashblock_sequence_fib_with_bal/stream/4fb_x_250tx                   1.00     55.5±0.17ms        ? ?/sec    1.03     57.0±1.02ms        ? ?/sec
flashblock_validation_launch_flashblock_sequence_fib_with_bal/stream/4fb_x_50tx                    1.00     13.4±0.12ms        ? ?/sec    1.00     13.4±0.12ms        ? ?/sec
flashblock_validation_launch_flashblock_sequence_world_id_like_bn254/stream/4fb_x_10tx             1.00    136.0±0.07ms        ? ?/sec    1.00    136.5±0.15ms        ? ?/sec
flashblock_validation_launch_flashblock_sequence_world_id_like_bn254/stream/4fb_x_12tx             1.00    163.0±0.11ms        ? ?/sec    1.00    163.3±0.22ms        ? ?/sec
flashblock_validation_launch_flashblock_sequence_world_id_like_bn254/stream/4fb_x_5tx              1.00     68.5±0.22ms        ? ?/sec    1.00     68.8±0.10ms        ? ?/sec
flashblock_validation_launch_flashblock_sequence_world_id_like_bn254_with_bal/stream/4fb_x_10tx    1.00     22.7±1.54ms        ? ?/sec    1.00     22.7±1.34ms        ? ?/sec
flashblock_validation_launch_flashblock_sequence_world_id_like_bn254_with_bal/stream/4fb_x_12tx    1.00     21.3±1.07ms        ? ?/sec    1.13     24.0±2.06ms        ? ?/sec
flashblock_validation_launch_flashblock_sequence_world_id_like_bn254_with_bal/stream/4fb_x_5tx     1.00     18.9±0.52ms        ? ?/sec    1.03     19.5±0.40ms        ? ?/sec
flashblock_validation_process_flashblock_eth_transfers/txs/1000                                    1.00     49.6±0.22ms        ? ?/sec    1.00     49.6±0.14ms        ? ?/sec
flashblock_validation_process_flashblock_eth_transfers/txs/50                                      1.05      2.8±0.42ms        ? ?/sec    1.00      2.7±0.02ms        ? ?/sec
flashblock_validation_process_flashblock_eth_transfers/txs/500                                     1.01     25.2±0.88ms        ? ?/sec    1.00     24.9±0.07ms        ? ?/sec
flashblock_validation_process_flashblock_eth_transfers_with_bal/txs/1000                           1.16    59.5±14.13ms        ? ?/sec    1.00     51.2±0.28ms        ? ?/sec
flashblock_validation_process_flashblock_eth_transfers_with_bal/txs/50                             1.10      3.5±0.53ms        ? ?/sec    1.00      3.2±0.05ms        ? ?/sec
flashblock_validation_process_flashblock_eth_transfers_with_bal/txs/500                            1.01     26.2±1.01ms        ? ?/sec    1.00     25.9±0.33ms        ? ?/sec
flashblock_validation_process_flashblock_fib/txs/1000                                              1.00     48.5±0.10ms        ? ?/sec    1.00     48.4±0.24ms        ? ?/sec
flashblock_validation_process_flashblock_fib/txs/50                                                1.02      2.7±0.18ms        ? ?/sec    1.00      2.6±0.03ms        ? ?/sec
flashblock_validation_process_flashblock_fib/txs/500                                               1.10     26.7±4.71ms        ? ?/sec    1.00     24.4±0.05ms        ? ?/sec
flashblock_validation_process_flashblock_fib_with_bal/txs/1000                                     1.03     50.7±2.79ms        ? ?/sec    1.00     49.3±0.22ms        ? ?/sec
flashblock_validation_process_flashblock_fib_with_bal/txs/50                                       1.01      3.2±0.37ms        ? ?/sec    1.00      3.2±0.04ms        ? ?/sec
flashblock_validation_process_flashblock_fib_with_bal/txs/500                                      1.02     25.9±1.96ms        ? ?/sec    1.00     25.5±0.99ms        ? ?/sec
flashblock_validation_process_flashblock_world_id_like_bn254/txs/10                                1.06     36.3±4.08ms        ? ?/sec    1.00     34.1±0.16ms        ? ?/sec
flashblock_validation_process_flashblock_world_id_like_bn254/txs/25                                1.00     84.7±1.17ms        ? ?/sec    1.00     84.6±0.12ms        ? ?/sec
flashblock_validation_process_flashblock_world_id_like_bn254/txs/50                                1.02    172.4±5.18ms        ? ?/sec    1.00    169.0±0.20ms        ? ?/sec
flashblock_validation_process_flashblock_world_id_like_bn254_with_bal/txs/10                       1.20      6.7±0.17ms        ? ?/sec    1.00      5.6±0.31ms        ? ?/sec
flashblock_validation_process_flashblock_world_id_like_bn254_with_bal/txs/25                       1.00      7.9±0.11ms        ? ?/sec    1.01      8.0±0.04ms        ? ?/sec
flashblock_validation_process_flashblock_world_id_like_bn254_with_bal/txs/50                       1.21     13.0±1.19ms        ? ?/sec    1.00     10.8±0.18ms        ? ?/sec

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant