derive and manage a fleet of builder keys from the entry key - #153
Open
pk910 wants to merge 27 commits into
Open
derive and manage a fleet of builder keys from the entry key#153pk910 wants to merge 27 commits into
pk910 wants to merge 27 commits into
Conversation
Builder keys are derived one EIP-2333 node below the operator's key, so index 0 is the entry key itself and higher indices cannot alias another participant's account path in a shared test setup. The registry owns derivation, per-key on-chain state resolved from the epoch snapshot in a single pass, and the usage history that makes a withdrawn key reusable instead of pushing the highest index up forever. Discovery scans past the target for keys used in an earlier run and stops after a full run of never-used indices; those scanned-but-unused indices are derived, not tracked, so the fleet view stays the size of the fleet.
Modules that held a single BLS signer now take the key registry and name the key each operation acts on: deposits, exits and top-ups take a key parameter, bid construction takes the key that signs, and the reveal service resolves its signer from the registry. Competitor bid comparisons exclude every managed key rather than one builder index, so a second key of ours can never read as our fiercest competitor. The deposit and exit commands gain --key-index. Behaviour is unchanged at the default target of one key: call sites use the primary key, which is internal index 0, which is the entry key.
Beacon nodes on the post-#624 beacon-API spec (e.g. the Lodestar on glamsterdam-devnet-7) reject publishExecutionPayloadEnvelope with 400 "Eth-Blob-Data-Included header is required": the spec renamed the body discriminator from Eth-Execution-Payload-Blinded to Eth-Blob-Data-Included (true = stateless SignedExecutionPayloadEnvelopeContents body). Bump go-eth2-client to a version that sends the new header (keeping the old one for beacon nodes that predate the rename).
A block's execution payload bid names the builder index it committed to, and that is the only thing that identifies which key won: several managed keys can bid the very same payload, so the block hash cannot say. BlockInfo now carries that index. The inclusion tracker resolves it to a key and refuses the win outright when it belongs to somebody else, rather than falling back to a key of ours — a validly-signed envelope for another builder's bid is rejected on chain while the slot is lost silently. The Builder API binds the same way at block submission. Payment accounting is per key, so a payment is charged to the key that owes it instead of draining a shared pool.
The lifecycle manager becomes a reconciler: it deposits keys until the managed count reaches the target, exits surplus keys down to it, and tops up whichever key fell below the threshold. Target changes wake it immediately instead of waiting out the idle tick. Everything is funded from one wallet, so a pass performs at most one transaction and the fleet ramps rather than flooding the deposit queue, whose fee grows with its length; the existing fee limit then backs the ramp off on its own. A wallet that cannot cover the next deposit reports once instead of failing a transaction per key. Early onboarding covers the whole target set: the deposits sit in the pending queue together and the fork transition converts them all, so the queue simulation now asks whether the batch's last entry survives.
The gossip rules ignore every bid after a builder's first for a slot, so one key can land only one bid however many candidates were built. Pairing each candidate with a distinct key is what makes them all propagate. The pairing is sticky per (slot, payload): an interval re-bid from another key would be a fresh first-seen bid, leaving the original lower bid as the one that actually reached the network. When there are fewer keys than candidates a key is reused rather than dropping the bid — bidding several candidates from one key remains a deliberate testing scenario. Selection strategies (round_robin, single, random, least_used) decide which keys cover the candidates, and bid_keys_per_slot caps how many bid at all. Balance is a preference, not a filter: an underfunded key is still offered when nothing else can cover the bid, since underfunded bids are a scenario worth testing. The Builder API picks the same way, sticky per (slot, parent tuple) so a polling proposer keeps seeing one builder.
Adds the builder-keys endpoints (list, target, per-key deposit/topup/exit) and a builder_keys SSE event carrying the whole set on every change, so the view stays live without polling. The target is written through the settings service rather than a second persistence path, which gets CLI/UI recency resolution and the audit log for free. An exit optionally decrements the target in the same request, and only after the exit landed — a failed exit must not shrink the fleet the operator asked for. Reading the key set works without lifecycle management: the keys are derived either way, only mutating them needs the manager. Bid attempts record the key index alongside the builder index, because a builder index is reused by other builders after an exit and the mapping is only reliable while the bid is being made.
The dashboard card gains an active/target key badge, an inline target editor that warns before a lowering exits keys, and fleet balance totals. Per-key operations move to a new Builder Keys page: one row per key with status, balances, usage and deposit/top-up/exit buttons. The exit confirmation names the consequences, because it cannot be undone, and offers to lower the target so no replacement is deposited. Bid popovers name the key that signed, and the stream now recognises a bid as ours by any managed key rather than the primary one — otherwise a second key's echoed bid renders as a competitor's.
The key strategy keys were declared but never wired into the field registry, so the API rejected them as unknown and neither the UI nor the state-db could reach them. Declaring a key without its field compiles and reads as complete, which is how it slipped through. The added test walks settings_keys.go and fails on any key missing from Fields(), plus checks flag keys are unique — two fields sharing a viper flag would resolve CLI changes against each other's last-seen value.
The balance service picks the top-up amount itself, falling back to the threshold when no amount is configured, so callers crediting the live balance with the configured amount credited the wrong number whenever those differ. It now returns what it deposited.
Every lifecycle transaction is serialized on the same funding key, so bringing a fleet up one confirmed transaction at a time costs a block per key. Deposits now sign consecutive nonces in one go and go out together, capped at ten per round. Each transaction is still resolved on its own, because sharing the funding key with other buildoor instances means any single nonce can be taken by a foreign transaction: the displaced one is rebuilt on a fresh nonce and retried while the ones that landed are left alone. Keys are claimed before submission so a batch cannot pick the same candidate twice, and released again when the batch never reached the chain.
… into pk910/multi-key
The scheduler ticks every 10ms while a bid submission is a network call taking tens of milliseconds. Marking the payload as bid only after the call returned let the ticks that landed mid-flight pass the interval check and gossip the same bid again — the beacon node rejects those as already known, and each one burns a key's single bid for the slot. Also stop exiting a lower key while higher-index ones are still on their way to active: those in-flight keys are the surplus, and exiting a usable key in their place burns a key we just paid for without shrinking the fleet once they register.
The key was pinned to the payload, so every re-bid of that payload came from the same key — and the gossip rules ignore a builder's later bids for a slot, so each escalation step was dropped as already known and the higher value never reached the network. A key is spent when one of its bids reaches the network, so that is what the slot now tracks. Every bid — an escalated re-bid of the same payload just as much as another candidate — claims a key that has not bid yet, and bidding stops once the fleet is exhausted rather than repeating a spent key. A submission that never made it out hands its key back. The escalation count moves to the payload too, so several candidates no longer inherit each other's step.
The key was claimed before the interval gate, so every scheduler tick that found nothing due still consumed one. At a 10ms tick and a 50ms interval the whole fleet was spent within a few ticks and the slot could no longer bid at all. The bid is now gated first and the key claimed only once it is due, with the payload's claim handed back when no key is left. Key readiness is checked against the escalated bid value rather than the configured minimum, so an underfunded key is not picked for a bid it cannot cover.
An exit takes a couple of epochs to show up as a withdrawable epoch in the beacon state. Until then the key still read active, so every reconcile pass picked it again and re-submitted the exit — paying the queue fee each time. On the devnet that was one redundant exit request every six seconds until the chain caught up.
The interval ladder is a single-key shape: one bid, wait, one more. With a fleet the useful shapes run from that ladder all the way to "every active key bids the moment the window opens", so a step now spends bid_keys_per_step keys instead of exactly one, each a value increment higher than the last. The submissions of a step are independent keys on independent payloads, so they go out concurrently — serialized they spread a slot's bids over tens of milliseconds each and waste the window.
A submission takes tens of milliseconds while the scheduler ticks every 10ms, so waiting on a step's submissions stalled the next step — and the next slot — for the duration of the slowest beacon call. The tick now dispatches and returns. Everything a later tick reads is already committed before any submission starts: the payload's interval claim and each key's claim are taken under one lock while planning the step. Shutdown drains the in-flight submissions separately.
A step's bids go out concurrently and reach the beacon node in arbitrary order, so giving each one a higher value made them race: every bid landing after a higher one came back BID_TOO_LOW. At 334 keys per step only a few dozen of a thousand bids survived. The escalation now counts steps, so one step bids one value from every key it spends and the next step bids one increment above it. A rejected bid also keeps its key spent. The node saw it and turned it down on merit, so retrying it unchanged only repeats the rejection — handing the key back had the slot spin through the whole fleet, reaching 1300 attempts for 1000 keys. Only a submission that never reached the node releases its key, told apart by the typed API error rather than its text.
Bidding a whole fleet makes rejection the normal outcome: the beacon node keeps only the best bid per (slot, parent), so every key beyond the first to arrive at a given value is turned down. At 600 keys that produced over 10k error lines per run, burying the transport failures that actually need attention. The per-attempt records on the slot result keep every rejection inspectable.
Two fixes the multi-key devnet run exposed: Bid values were escalated per interval step, so every key of a step signed the same value. Gossip only forwards the highest bid seen for a (slot, parent) tuple, so all but the first to arrive were dropped as too low -- spending 300 keys to propagate one bid. Each bid now lands one increase above the last. The lifecycle manager's deposit-pending callback pulled the p2p bidder's registration state back to pending on every deposit batch. That is single key semantics: a fleet ramping toward its target deposits continuously, so bidding stayed suppressed for the whole ramp even with keys already active. It now only applies when no key of the fleet is active.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on #152 (
pk910/reorg-handling) — review after that one; merging #152 rebases this to a pure multi-key diff.Gloas gossip only accepts the FIRST bid a builder publishes for a slot — a single-key builder gets exactly one shot per slot, which makes bid escalation, multi-candidate bidding (#152) and competitive-auction testing impossible. This PR turns buildoor's single BLS identity into a managed fleet of builder keys derived from the operator's entry key, with the lifecycle manager reconciling the on-chain fleet against a target count and the bidder spending a fresh key per bid.
Default behaviour is unchanged:
--builder-keys-target 1(the default) is the entry key alone.Builder key registry (
pkg/builder_keys/)The identity dependency of every module that used to hold a single
*signer.BLSSigner:--builder-privkeyor--builder-mnemonic+--builder-key-index. Internal index 0 IS the entry key; index n ≥ 1 isderive_child_SK(entry_sk, n)— one EIP-2333 node deeper than any participant's account path, so fleets are collision-free between builders sharing a mnemonic.KeyIndexis our derivation index (stable forever);BuilderIndexis the beacon registry index, assigned at deposit and reused by other builders after an exit.kv_storebuilder_keysnamespace, so a key deposited in an earlier run is recognised before the beacon state confirms it and exited indices are reused. A persisted pubkey disagreeing with the derived one is a hard startup error (the entry key changed). Startup discovery scans past the target for previously used keys, stopping afterbuilder_keys.discovery_gapunused indices.SelectForBid):round_robin(default) /single/random/least_used, excluding keys already committed for the slot. Balance is a preference, not a filter — underfunded keys sort last but are still offered, since deliberately underfunded bids are a scenario worth testing.Lifecycle manager becomes the fleet reconciler (
pkg/lifecycle/)builder_keys.target_count: deposits below it (auto_deposit), exits surplus above it (auto_exit, highest index first, skipping keys with pending payments the chain would silently ignore), tops up whichever key fell below the threshold.deposit_max_feebacks the ramp off on its own. Deposits go out as batched transactions (pkg/wallet/batch.go).OnChangewiring.Multi-key bidding
(slot, parent)tuple, so bids sharing a value cannot all propagate no matter how many keys sign them.epbs.bid_keys_per_stepcontrols how many keys bid a payload per interval step (each one increment higher; 1 = walk the fleet up the price ladder, 0 = spend every remaining key at once);epbs.bid_keys_per_slotcaps distinct keys per slot. One step's submissions are dispatched concurrently without blocking the 10ms scheduler tick, and a payload's bid slot is claimed before submission so concurrent steps can't double-spend a key.builder_api.key_strategy, empty = follow the ePBS strategy).API / UI
New Builder Keys view (fleet table with status, balances, usage, deposit/exit/top-up actions),
/api/buildoor/builder-keysendpoints (list/deposit/exit/topup, auth + audit + swagger),builder_keysSSE updates, and akeysCLI helper wiring. The builder-info panel shows the fleet summary.Config surface
builder_keys.*(target_count,max_index,discovery_gap,auto_deposit,auto_exit— auto-exit is irreversible, an exited key can never be reactivated),epbs.key_strategy,epbs.bid_keys_per_slot,epbs.bid_keys_per_step,builder_api.key_strategy; all mutable at runtime except the discovery gap.Notes
Eth-Blob-Data-Includedenvelope-header fix (go-eth2-client bump) via a merge of send the Eth-Blob-Data-Included header on envelope submission #150; that diff disappears once send the Eth-Blob-Data-Included header on envelope submission #150 lands on main and the stack rebases.Devnet findings
Validated on a six-client Gloas devnet (lodestar/nethermind, lighthouse/geth, teku/besu,
prysm/reth, lighthouse/erigon, grandine/ethrex). Two bugs this surfaced are fixed here:
every bid after the first to arrive is dropped as too low. Escalating per bid took
acceptance from ~10% (64/606) to ~99% (253/256).
the p2p bidder's registration state back to
pendingon every deposit batch — single-keysemantics. With a fleet ramping toward its target that suppressed bidding for the entire
ramp despite active keys. It now only applies when no key of the fleet is active.
Fleet scale checked at 1000 keys (~10 min ramp, ~100 MB RSS, <10% CPU) and 300 keys bidding
120-300 escalating bids per slot with a stable mesh.
Testing
go build ./.../go test ./...pass — new tests for derivation (EIP-2333 vectors), registry state/persistence/discovery, selection strategies, per-key scheduler behaviour, payment binding, batch deposits, the pending-deposit simulation and the new API handlers; sharedkeyset_test.gofixtures across the bidder/builderapi/payload_bidder packages.