chore: merge shadow branch fixes and improvements to main

[GrapeBaBa]

Summary

Cherry-pick all non-shadow-specific commits from the shadow branch to main. These are fixes, refactors, and features that accumulated on shadow while it was the active development branch.

Key changes:

• xev Dynamic API for runtime io_uring/epoll detection (beneficial for all platforms)
• Store refactor: key by AttestationData (refactor: update store to key by AttestationData #656)
• Multiple bug fixes: checkpoint sync, subnet subscription, pre-finalized attestations, segfaults, etc.
• Devnet3 metrics, key-manager consolidation, performance improvements
• Documentation updates

Not included (shadow-only):

• rust/patch/quinn-udp/ (Shadow fallback UDP)
• shadow/ directory (removed, now in lean-quickstart)
• Intermediate Shadow config commits

After this merges, the shadow branch can be rebased onto main with minimal diff (only quinn-udp patch).

Test plan

• CI passes on all existing tests
• Build succeeds on Linux and macOS

[GrapeBaBa]

Wrong approach — included unrelated commits. Will recreate with only shadow-specific changes.

[Copilot]

Pull request overview

This PR cherry-picks a large set of non-shadow-specific fixes/refactors/features from the former shadow development branch onto main, including runtime I/O backend detection, a fork-choice/store refactor around AttestationData keys, and multiple operational/CLI/metrics improvements.

Changes:

• Refactors gossip-signature + aggregated-payload storage to be keyed by AttestationData (nested per-validator inner maps) and updates aggregation/block-production paths accordingly.
• Switches xev usage to the Dynamic API with runtime backend detection (io_uring/epoll probing on Linux) and propagates formatting changes ({f}) across logging/fixtures.
• Adds test key infrastructure (new test-keys submodule, key loading helpers, and a tools keygen command), plus metrics and CLI/database recovery improvements.

Reviewed changes

Copilot reviewed 40 out of 40 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
resources/zeam.md	Adds links to LeanSpec client documentation.
resources/poseidon.md	Documents Poseidon2 SSZ hashing build flag and design notes.
README.md	Mentions Poseidon2 option and adds build troubleshooting notes.
pkgs/types/src/state.zig	Changes pre-finalized attestation handling (skip vs error) + updates test.
pkgs/types/src/lib.zig	Updates public exports to match new attestation/signature aggregation API.
pkgs/types/src/block.zig	Introduces SignaturesMap keyed by AttestationData and AggregateInnerMap; updates proto types.
pkgs/types/src/block_signatures_testing.zig	Removes legacy aggregation test suite tied to old SignatureKey design.
pkgs/tools/src/main.zig	Adds keygen command to generate XMSS test validator keys + manifest.
pkgs/state-transition/src/transition.zig	Updates PQ signature verification metrics and invalid/valid counters.
pkgs/state-transition/src/mock.zig	Updates mock chain generation to use new signature map + inner-map aggregation.
pkgs/spectest/src/runner/fork_choice_runner.zig	Updates fixtures runner formatting and aggregated-payload storage API usage.
pkgs/spectest/src/json_expect.zig	Updates fixture error formatting to use {f} formatting.
pkgs/node/src/utils.zig	Switches to xev.Dynamic and adds detectBackend() runtime probe.
pkgs/node/src/tree_visualizer.zig	Aligns fork-choice JSON output shapes with leanSpec + adds test coverage.
pkgs/node/src/testing.zig	Uses xev.Dynamic and calls detectBackend() during test context init.
pkgs/node/src/node.zig	Adds subnet subscription controls, sync recovery logic, cached-block followups, and formatting updates.
pkgs/node/src/lib.zig	Re-exports detectBackend from utils.
pkgs/node/src/forkchoice.zig	Refactors signature/payload store to AttestationData keys; new proposal attestation selection algorithm; pruning/metrics updates.
pkgs/node/src/constants.zig	Adds periodic sync status refresh interval constant.
pkgs/node/src/clock.zig	Switches to xev.Dynamic and handles timer cancelation cleanly.
pkgs/node/src/chain.zig	Updates block production to new proposal attestation API; improves gossip missing-root propagation; memory-safety errdefers.
pkgs/network/src/mock.zig	Uses xev.Dynamic, backend detection in tests, and minor loop refactor.
pkgs/network/src/interface.zig	Uses xev.Dynamic; updates GossipTopic formatting signature and log format specifiers.
pkgs/network/src/ethlibp2p.zig	Uses xev.Dynamic and updates log formatting specifiers.
pkgs/metrics/src/lib.zig	Renames/adds PQ signature metrics, adds fork-choice store gauges and committee aggregation histogram; adjusts buckets.
pkgs/key-manager/src/lib.zig	Adds pre-generated key loading, ownership tracking, shared key loading helper, and signing metric updates.
pkgs/database/src/rocksdb.zig	Improves finalized-state recovery logs and corruption diagnostics.
pkgs/cli/test/fixtures/config.yaml	Adds ATTESTATION_COMMITTEE_COUNT to fixture config.
pkgs/cli/src/node.zig	Adds DB wipe/reopen helper, checkpoint-vs-db anchor selection logic, subnet/committee-count options parsing, and key loading helper use.
pkgs/cli/src/main.zig	Uses xev.Dynamic, calls detectBackend() at startup, and uses shared key loading helper.
pkgs/api/src/events.zig	Updates proto block construction to include proposer_index.
pkgs/api/src/event_broadcaster.zig	Updates proto block construction to include proposer_index.
Dockerfile	Switches release build optimization to ReleaseSafe.
build.zig.zon	Sets minimum Zig version to 0.15.0.
build.zig	Injects test_keys_path, wires build_options into key-manager, updates tools deps, and forces ReleaseFast for riscv32 targets.
.gitmodules	Adds test-keys submodule.
.github/workflows/ci.yml	Ensures submodules are checked out; switches build optimization to ReleaseSafe.
.github/workflows/auto-release.yml	Switches release builds to ReleaseSafe.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

getProposalAttestationsUnlocked iterates over latest_known_aggregated_payloads without taking signatures_mutex, but that map is mutated under signatures_mutex by storeAggregatedPayload / acceptNewAttestationsUnlocked / aggregateCommitteeSignaturesUnlocked. Iterating without the same lock can race with concurrent writers and lead to iterator invalidation / memory corruption. Consider either (a) taking signatures_mutex while collecting the relevant payload entries into a local list (keys + cloned proofs) and then releasing it before the STF loop, or (b) protecting payload maps with the main mutex consistently (writers + readers).

[Copilot]

SignaturesMap.getPtr takes self: *const SignaturesMap but returns a mutable ?*InnerMap, which breaks const-correctness (callers can mutate inner maps through a const reference). This should either take self: *SignaturesMap or return ?*const InnerMap (and provide a separate mutable accessor when needed).

[Copilot]

SignaturesMap.get returns an InnerMap by value (copying the hash map struct). Even if callers treat it read-only, this makes it easy to accidentally mutate/deinit a copy and double-free internal allocations. Prefer exposing only pointer-based accessors (getPtr / getConstPtr) so ownership/lifetime is unambiguous.

[Copilot]

After storeAggregatedPayload was refactored to no longer take validator_ids, this block still allocates/fills validator_ids but never uses it. Zig will treat this as an unused local and fail compilation. Remove the allocation/loop (or use it if still needed).

[Copilot]

wipeAndReopenDb builds rocksdb_path via "{s}/rocksdb" and then calls std.fs.deleteTreeAbsolute(rocksdb_path). If database_path is a relative path (common for CLI flags/defaults), deleteTreeAbsolute will error because the path is not absolute. Use std.fs.cwd().deleteTree(rocksdb_path) for relative paths, or convert to an absolute path first (e.g., realpathAlloc) before calling deleteTreeAbsolute.