Metrics to add tracker

[dapplion]

Ready to execute

• Track UnhandledPromiseRejectionWarning count with metric  #2483
• Add op pool metrics #2984
• Add async event loop lag metric #3033
• Add total synced blocks metrics #3112
• Use libp2p metrics #3115
• Improve REST API metrics #3205
• Metric to track sync committee members #3413
• Add state cloneCount metric #3680
• Eth1 + Builder API Metrics #3946
• Add reqResp request time metrics #4122 ReqResp request times
• Add reorg and parent distance metrics #4123 Reorg depth histogram
• Add reorg and parent distance metrics #4123 Distance to parent block histogram
• Add proposerBalanceDiff metric #4124 Histogram of balance increase for block proposer
• Add bls.aggregatedPubkeys metric #4125 Total signatures aggregated (optional histogram for time spent aggregating)
• Add JSON RPC client metrics #4127 JSON RPC client metrics
• From Lighthouse: Block processing detailed metrics #4273
• From Lighthouse: Block production detailed metrics #4274
• From Lighthouse: Add ForkChoice metrics #2558
• From Lighthouse: Add op pool metrics #2984
• From Lighthouse: Add validator monitor metrics #2473

Metric ideas

Misc

Once properly defined, convert to issue or pull request and move above to "Ready to execute"

• Improve state.clone() metrics, what we really care about is before doing state-transitions if the state has cache or not.
• Add more sync metrics to measure efficiency: # blocks downloaded to process # of blocks. To count duplicated downloads indirectly.
• Discv5 ping times
• Gossipsub received blocks
• Blocks in last epoch, to compute percent of orphaned
• More HTTP metrics?

From Lighthouse

Reviewed this file and select interesting metrics to add
https://github.com/sigp/lighthouse/blob/stable/beacon_node/beacon_chain/src/metrics.rs

Block Delay Metrics

• beacon_block_imported_slot_start_delay_time: Duration between the start of the blocks slot and the current time when it was imported.
• beacon_block_head_slot_start_delay_time: Duration between the start of the blocks slot and the current time when it was as head.
• beacon_block_head_slot_start_delay_exceeded_total: Triggered when the duration between the start of the blocks slot and the current time will result in failed attestations. feat: add setHeadAfterFirstInterval metric #5559

Persisting BeaconChain components to disk

• beacon_persist_head: Time taken to persist the canonical head
• beacon_persist_op_pool: Time taken to persist the operations pool - Does Lighthouse keep all ops in memory?
• beacon_persist_eth1_cache: Time taken to persist the eth1 caches

Chain Head

• beacon_update_head_seconds: Time taken to update the canonical head - Review what does it refer too

[g11tech]

assign please

[g11tech]

@dapplion these looks like good metrics to pick up next ?

[dapplion]

@dapplion these looks like good metrics to pick up next ?

I think the next interesting group to add would be around block production. Note that lighthouse has very detailed timings on each step of the process. Our approach to block production is more simple so maybe a single histogram around the entire thing would suffice. We are already kind of tracking it via the API response times, but it would be nice to track it anyway.

• beacon_block_production_requests_total: Count of all block production requests
• beacon_block_production_successes_total: Count of blocks successfully produced
• beacon_block_production_seconds: Full runtime of block production

At the same time it would be quite a priority to have metrics in our regen / state cache: #2153. Let's discuss details in that issue if you are interested.

[dapplion]

Keeping open since there are some metrics we want to implement soonish too

[philknows]

Adding some ideas to this from @wemeetagain discussion in Columbia that I found in my notes:

• Block size metrics (due to 4844 blobs) to test performance
• Number of operations per block
• Metrics on exits, slashings, deposits.