build multiple candidate payloads per slot for reorg and payload-miss testing - #152
Open
pk910 wants to merge 12 commits into
Open
build multiple candidate payloads per slot for reorg and payload-miss testing#152pk910 wants to merge 12 commits into
pk910 wants to merge 12 commits into
Conversation
…tatus and build-parent candidates - pkg/chain/headtracker.go: shared block-by-root ancestry cache, reorg detection with head-change events, Gloas payload reveal status (payload-available events + child-block evidence + reveal-deadline fallback), and build-parent candidate resolution (parent/grandparent x full/empty tuples incl. EL parent gas limit / block number) - subscribe the beacon chain_reorg SSE topic as a cross-check signal - BlockInfo carries the committed gas limit (bid or embedded payload) and the pre-Gloas execution block number - parse PAYLOAD_DUE_BPS from the chain spec (default 5000) - refetch cached epoch state when a head event's duty dependent root proves a reorg crossed the epoch boundary; fix unlocked currentEpoch read in RefreshBuilders - inclusion tracker resolves ancestry blocks through the shared cache
…ent changes - the attributes cache keys events by (parent_block_root, parent_block_hash): reorgs and Gloas full/empty parent flips re-emit attributes for the same proposal slot with a different parent, and every variant is retained (last-writer-wins per variant, newest overall as the slot's latest) - a later attributes event with a different parent aborts the in-flight build for the slot and rebuilds on the new parent instead of being silently ignored - sanitize attributes against the chain view before building: redirect references to a known-withheld execution payload to the last actually built block, and backfill missing parent block numbers (several clients omit them under Gloas) - missing-attributes fallback resolves the target slot's proposer from cached duties instead of copying the source slot's, and warns when a synthesized event crosses an epoch boundary (stale prev_randao)
A slot can now produce several payloads, one per build-parent candidate:
parent_full (normal), parent_empty (Gloas payload miss: head block on its
execution parent), grandparent_full (deliberate reorg of the head block)
and grandparent_empty. Which candidates build is configurable per key
(auto/always/never; auto derives from the parent payload's reveal status
and head-vote weakness) via config, CLI flags and the mutable settings
registry.
- the build pass resolves the candidate set at build start from policy +
chain view + received attribute variants; missing variants are
synthesized (empty-parent withdrawals from the parent slot's attributes
per the unchanged expected-withdrawals rule; grandparent candidates
re-target the parent slot's attributes, refused across epoch
boundaries); received variants on unknown branches still build
- sequential builds run speculative candidates first and the canonical one
last so the EL head ends on the canonical chain; build.parallel opts
into concurrent engine builds, build.speculative_build_time_ms shortens
speculative builds
- variants arriving after the build pass activate additional candidate
builds until slot end instead of being dropped
- payload cache, build events and slot results are candidate-aware: the
cache keys payloads by parent tuple with priority-based primary
selection, slot results carry a builds list next to the primary build,
and payload artifacts get per-slot indices with candidate metadata
(legacy artifacts read as index 0; /payload/{index} endpoint added)
- attribute sanitization re-sources withdrawals when redirecting a
full-parent reference to a withheld payload's execution parent
- flag-gated bid gas-limit enforcement (build.enforce_bid_gas_limit,
default off): adjust the built payload to the exact gossip-legal gas
limit derived from the actual EL parent payload
- engine fcU carries the resolved proposer fee recipient instead of the
builder default; fix mislabeled parent_hash log field
- p2p bidding picks its candidate payload at bid time from the live epbs.bid_candidate setting: auto (match the chain view's head and payload status; sticky per slot unless epbs.bid_candidate_switch), a forced candidate key, or all (gossip one bid per built candidate — multi-parent gossip testing). Single-bid dedup is per payload, so a candidate switch or rebuild bids again - bidding reopens for a slot when the block that closed it is reorged out; per-slot bid state and the competitor tracker prune on epoch transitions - competitor-high comparison only considers bids committing to the same execution parent - builder-api bid requests are answered from whichever candidate payload matches the requested (parent_hash, parent_root) — governed by builder_api.serve_candidates (all / canonical_only / key list) — with optional on-demand building (builder_api.on_demand_build) for legal parent tuples no candidate covers; requests for a chain-view-illegal tuple still fail with 400, everything else degrades to 204 - legacy getHeader serves the candidate payload matching the requested parent hash
- a reveal request for an already scheduled slot re-binds the reveal when our payload is re-included under a different beacon block after a reorg: the envelope is rebuilt and re-signed for the new root (the signature covers the block root, so the old envelope cannot be reused; gossip seen-caches are per builder + block root). Gated by reveal.rebind_on_reorg (default on) and verified against the chain view - an orphaned winning block clears the payload's won marker (so a re-inclusion is detected and counted again) and flags the pending payment as disputed, excluding it from the pending total; the block returning to the canonical chain restores it. Locally settled payments (already revealed) are only logged — the on-chain payment quorum owns the final outcome
…re WebUI - the plan's build category gains a candidates map (candidate key -> auto/always/never) merged with the global policy into the frozen snapshot's complete candidate_modes; bid plans gain bid_candidate, builder-api plans gain serve_candidates — all consumers read the frozen per-slot values - SSE build events carry the candidate key; the slot timeline tracks every candidate build: the primary payload keeps the classic build line (colored by candidate), other candidates render as thin stacked mini-bars, and a badge popover lists each candidate's parent, timing and outcome; the legend shows the candidate palette - slot results carry the per-candidate builds list next to the primary build, each with its payload artifact index - the slot edit modal adds per-candidate build policy selects (single-slot editing), a bid-candidate select and a serve-candidates select; plan cells show a candidate-override chip
…starved The per-slot LastBidTime gate let the first candidate payload's bid throttle every other candidate in the same tick: with bid_candidate=all and interval bidding only one candidate ever gossiped. Interval throttling and single-bid dedup now key on the payload (BidPayloads maps payload hash -> last bid time), so each candidate bids on its own schedule. Found live on a glamsterdam-devnet-7 devnet run.
…ilder UI Devnet feedback on the multi-candidate build pass: - candidates now build concurrently by default, so all of them complete within the slot's designated build window instead of serializing into it. Serialized builds remain available (build.parallel=false) and run the canonical candidate first so it keeps its scheduled start; the speculative build time only applies there - fix candidate builds cancelling each other: parent_empty and grandparent_full extend the same execution parent and differ only in the beacon parent, so the active-build key needs the full parent tuple. Retiring a build now also only clears its own entry - resolve the forkchoice safe/finalized hashes once per head change in the head tracker instead of per build (each build paid four beacon-API round trips), and keep candidate resolution cache-only: the execution block number is backfilled from cached envelope metadata, prefetched when a payload-available event arrives, never fetched on the hot path - the build-candidate policy is now editable in the UI, as a section of the Payload Builder card (per-candidate auto/always/never, weak-head threshold, parallel toggle, speculative build time, gas-limit enforcement) - slot graph: the primary build line spans the primary candidate's own build instead of stretching across every candidate as they complete; the other candidates' mini-bars are clickable and sit inside the row, and the candidate badge renders below the build line so it is not clipped when building starts before the slot
The stacked per-candidate mini-bars cluttered the slot row for little value. The graph now draws a single build line again (the primary candidate's, colored by candidate), and the payload-created dot opens a tabbed callout instead: one tab per built candidate, ordered parent → grandparent, each showing that payload's own hash, value, block number, gas and contents, with its own JSON/SSZ download. The payload artifact endpoint takes an optional ?candidate= selector for this, resolving the slot's payload artifact index from the stored candidate metadata (404 when the slot built no payload for it).
Which payload an event actually used was invisible in the UI, and the payload callouts showed no parent context. - bid, getExecutionPayloadBid, getHeader and won-block callouts now name the candidate payload they committed to, matched by block hash (shown only when the slot built more than one candidate) - payload callouts show the parent tuple the payload extends: the beacon parent root with its slot and the execution parent hash with its block number, per candidate tab - payload_ready carries the parent block root, its slot and the execution parent's block number; the beacon parent's slot is resolved from the head tracker's cache so event assembly never blocks
CI runs gofmt with -s (simplify); these files only satisfied plain gofmt. Alignment of const and struct-field blocks, one import ordering, and three doubled blank lines. No behaviour change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A Gloas builder that only ever builds on "the head block and its revealed payload" has nothing to bid when the interesting cases happen: the parent payload is withheld, the head is contested, or we deliberately want to reorg the head block. This PR makes buildoor build several candidate payloads per slot — one per parent tuple — and lets every consumer (p2p bids, Builder API, reveals) pick the right one at request time.
The four candidate keys:
parent_fullparent_emptygrandparent_fullgrandparent_emptyWhat's in here
Chain view / head tracker (
pkg/chain/headtracker.go): buildoor's own canonical-chain oracle — current head, bounded block-by-root ancestry cache, per-block Gloas payload reveal status (payload-available events + child-block evidence +PAYLOAD_DUEfallback), reorg detection with depth/common-ancestor (the beaconchain_reorgSSE topic subscribed as a cross-check), and build-parent candidate resolution. Also refetches epoch stats when a head event's duty dependent root proves a reorg crossed the epoch boundary.Candidate builds (
pkg/payload_builder/): per-candidate policyauto/always/neverviabuild.candidate_*settings —autobuilds speculative candidates on chain signals (parent payload not revealed, head-vote participation belowbuild.auto_weak_head_pct). Payload-attributes are cached per (slot, parent_root, parent_hash) variant, sanitized against the chain view (Grandine's wrong miss-case parent is corrected, missing parent numbers backfilled) and synthesized for candidates no client emits attributes for (grandparent candidates re-target the parent slot's attributes; refused across epoch boundaries). Sequential builds run speculative candidates first and canonical last;build.parallelopts into concurrent engine builds,build.speculative_build_time_msgives speculative builds their own EL build time.Request-time selection: p2p bids commit to a candidate per
epbs.bid_candidate(auto= match the live chain view, sticky per slot unlessepbs.bid_candidate_switch;all= deliberate multi-parent gossip). The Builder API serves whichever candidate matches the requested parent (builder_api.serve_candidatespolicy) and can build a missing legal parent on the fly (builder_api.on_demand_build). Interval bids are throttled per payload so multi-candidate bidding doesn't starve any one candidate.Reorg-aware wins and reveals: reveals re-bind (rebuilt, re-signed envelope) when a won payload is re-included under a different block root (
reveal.rebind_on_reorg); orphaned wins dispute pending payments and clear won markers until the block returns — payment tracking follows the inclusion verdict instead of the first sighting.Gas-limit enforcement: flag-gated
build.enforce_bid_gas_limitadjusts built payloads to the exact bid-gossip-legal gas limit (EL parent gas limit stepped toward the proposer target) for ELs that ignore the target gas limit attribute.Action plan + WebUI: per-slot
build.candidatesoverrides in the action plan (resolved into the frozen plan), a candidate policy section in the builder config panel, candidate payloads as tabs in the payload-created callout, and the committed candidate + parent tuple shown in slot callouts. Slot results and artifacts store one payload per candidate.Config surface
New
build.*section (candidate_parent_full/empty,candidate_grandparent_full/empty,parallel,speculative_build_time_ms,auto_weak_head_pct,enforce_bid_gas_limit), plusepbs.bid_candidate,epbs.bid_candidate_switch,builder_api.serve_candidates,builder_api.on_demand_buildandreveal.rebind_on_reorg— all registered as mutable settings and exposed as CLI flags. Defaults keep today's behaviour (canonical candidate only).Testing
go build ./.../go test ./...pass; new unit tests for the head tracker, attributes variants/synthesis, candidate resolution, scheduler throttling and the action plan candidate overrides