derive and manage a fleet of builder keys from the entry key

CLAUDE.md

@@ -333,10 +333,51 @@ npm run clean
      participation opens reveal vote gates late or never (withheld at slot
      end)
 
-4. **Lifecycle Manager** (`pkg/lifecycle/`)
-   - Builder registration on beacon chain
-   - Balance monitoring and auto top-ups
-   - Deposit and exit operations
+3b. **Builder Key Registry** (`pkg/builder_keys/`) — the managed set of builder
+   BLS keys and the identity dependency of every module that used to hold a
+   single `*signer.BLSSigner`.
+   - **Derivation**: keys come from the operator's ENTRY key (`--builder-privkey`
+     or `--builder-mnemonic` + `--builder-key-index`). Internal index 0 IS the
+     entry key; index n ≥ 1 is `derive_child_SK(entry_sk, n)` — one EIP-2333 node
+     deeper than any other participant's account path, which is what makes the
+     set collision-free with other builders sharing a mnemonic. Derivation is
+     cached for the process lifetime (`signer.DeriveInternalKey`).
+   - **Two index spaces, never conflated**: `KeyIndex` is our derivation index
+     (stable forever); `BuilderIndex` is the beacon registry index, assigned at
+     deposit and REUSED by other builders after an exit.
+   - **State** per key (`Status`: unused / depositing / pending / active /
+     exiting / exited / withdrawn) is resolved from the epoch snapshot in ONE
+     pass over `chainSvc.GetBuilders()`, so monitoring hundreds of keys costs the
+     same beacon query as one. Only transactions scale with the fleet size.
+   - **Usage history** (`kv_store` namespace `builder_keys`) survives restarts so
+     a key deposited in an earlier run is recognised before the beacon state
+     confirms it, and a key whose pubkey has left the registry is reused instead
+     of pushing the highest index up forever. A persisted pubkey that disagrees
+     with the derived one is a hard startup error (the entry key changed).
+   - **Discovery** scans past the target for keys we used before and stops after
+     `builder_keys.discovery_gap` never-used indices. Scanned-but-unused indices
+     are derived, not tracked, so the fleet view is the size of the fleet.
+   - **Selection** (`SelectForBid`) orders active keys per strategy
+     (`round_robin` default, `single`, `random`, `least_used`) excluding keys
+     already committed for the slot. Balance is a PREFERENCE, not a filter:
+     underfunded keys sort last but are still offered when nothing else can
+     cover the bid — deliberately underfunded bids are a scenario worth testing.
+
+4. **Lifecycle Manager** (`pkg/lifecycle/`) — the fleet reconciler
+   - Keeps the managed key count at `builder_keys.target_count`: deposits keys
+     below it (`auto_deposit`), exits surplus keys above it (`auto_exit`,
+     highest index first, skipping keys with pending payments the chain would
+     silently ignore), and tops up whichever key fell below the threshold
+   - At most ONE lifecycle transaction per reconcile pass: they all go through
+     the single funding wallet and each waits for its receipt, so the fleet
+     ramps instead of flooding the EIP-8282 deposit queue (whose fee grows with
+     its length). `deposit_max_fee` then backs the ramp off on its own; a wallet
+     that cannot cover the next deposit reports once instead of failing per key
+   - Target changes wake the reconciler immediately (`Manager.Reconcile()`,
+     wired from the settings `OnChange`)
+   - Early onboarding covers the WHOLE target set: the pre-Gloas deposits sit in
+     the pending queue together and the fork transition converts them all, so
+     `pending_deposit_sim.go` asks whether the batch's LAST entry survives
    - Optional component (only active with `--lifecycle` flag)
 
 4b. **Slot Results Tracker** (`pkg/slot_results/`) — generic per-slot outcome history
@@ -451,7 +492,25 @@ Configuration is managed via:
 - Environment variables (auto-loaded by viper)
 
 Key config sections:
-- **Builder keys**: `--builder-privkey` (BLS), `--wallet-privkey` (ECDSA)
+- **Builder keys**: `--builder-privkey` (BLS) or `--builder-mnemonic` +
+  `--builder-key-index` (the ENTRY key), `--wallet-privkey` (ECDSA)
+- **Managed key set**: `--builder-keys-target` (keys kept registered and
+  funded; default 1 = the entry key alone, byte-identical to single-key
+  behaviour), `--builder-keys-max-index` (derivation cap, default 1000),
+  `--builder-keys-discovery-gap` (unused indices ending the startup scan,
+  default 100), `--builder-keys-auto-deposit` / `--builder-keys-auto-exit`
+  (both default true; auto-exit is irreversible — an exited key cannot be
+  reactivated). All but the discovery gap are mutable via
+  `builder_keys.*` settings keys
+- **Key selection**: `--epbs-key-strategy` (round_robin | single | random |
+  least_used), `--epbs-bid-keys-per-slot` (cap per slot, 0 = no cap beyond the
+  fleet), `--epbs-bid-keys-per-step` (keys bidding a payload per interval step,
+  each one increment higher; 1 = walk the fleet up the ladder, 0 = spend every
+  remaining key at once), `--builder-api-key-strategy` (empty = follow the ePBS
+  strategy). A key is SPENT once one of its bids reaches the network — the
+  gossip rules ignore a builder's later bids for a slot — so every bid, an
+  escalated re-bid of the same payload included, takes a key that has not bid
+  yet, and the submissions of one step go out concurrently
 - **Clients**: `--cl-client`, `--el-engine-api`, `--el-rpc`
 - **Schedule**: `--schedule-mode` (all/every_nth/next_n), `--schedule-every-nth`, `--schedule-next-n`
 - **ePBS timing**: `--build-start-time`, `--epbs-bid-start`, `--epbs-bid-end`

cmd/deposit.go

@@ -11,7 +11,6 @@ import (
 	"github.com/ethpandaops/buildoor/pkg/lifecycle"
 	"github.com/ethpandaops/buildoor/pkg/rpc/beacon"
 	"github.com/ethpandaops/buildoor/pkg/rpc/execution"
-	"github.com/ethpandaops/buildoor/pkg/signer"
 	"github.com/ethpandaops/buildoor/pkg/wallet"
 )
 
@@ -53,10 +52,17 @@ var depositCmd = &cobra.Command{
 		}
 		defer rpcClient.Close()
 
-		// Initialize BLS signer (raw hex key or mnemonic-derived)
-		blsSigner, err := signer.NewBuilderSigner(cfg.BuilderPrivkey, cfg.BuilderMnemonic, cfg.BuilderKeyIndex)
+		// Initialize the managed builder key set and select the key to deposit for
+		registry, err := newKeyRegistry(cfg, logger)
 		if err != nil {
-			return fmt.Errorf("invalid builder key: %w", err)
+			return err
+		}
+
+		keyIndex, _ := cmd.Flags().GetUint64("key-index")
+
+		key, err := registry.Key(keyIndex)
+		if err != nil {
+			return err
 		}
 
 		// Initialize wallet
@@ -92,7 +98,7 @@ var depositCmd = &cobra.Command{
 		defer chainSvc.Stop() //nolint:errcheck // cleanup
 
 		// Check if builder already registered
-		pubkey := blsSigner.PublicKey()
+		pubkey := key.Pubkey()
 
 		builderInfo := chainSvc.GetBuilderByPubkey(pubkey)
 		if builderInfo != nil {
@@ -110,25 +116,26 @@ var depositCmd = &cobra.Command{
 		timeout, _ := cmd.Flags().GetDuration("timeout")
 
 		// Initialize lifecycle manager
-		lifecycleMgr, err := lifecycle.NewManager(cfg, clClient, chainSvc, blsSigner, w, logger)
+		lifecycleMgr, err := lifecycle.NewManager(cfg, clClient, chainSvc, registry, w, logger)
 		if err != nil {
 			return fmt.Errorf("failed to initialize lifecycle manager: %w", err)
 		}
 
 		logger.WithFields(map[string]any{
+			"key":         key.String(),
 			"pubkey":      fmt.Sprintf("%x", pubkey[:8]),
 			"amount_gwei": amount,
 		}).Info("Creating builder deposit")
 
 		// Ensure builder is registered
-		if err := lifecycleMgr.EnsureBuilderRegistered(ctx); err != nil {
+		if err := lifecycleMgr.EnsureBuilderRegistered(ctx, key); err != nil {
 			return fmt.Errorf("failed to ensure builder registered: %w", err)
 		}
 
 		if waitForInclusion {
 			logger.Info("Waiting for registration...")
 
-			if err := lifecycleMgr.WaitForRegistration(ctx, timeout); err != nil {
+			if err := lifecycleMgr.WaitForRegistration(ctx, key, timeout); err != nil {
 				return fmt.Errorf("registration wait failed: %w", err)
 			}
 
@@ -147,6 +154,7 @@ func init() {
 	rootCmd.AddCommand(depositCmd)
 
 	depositCmd.Flags().Uint64("amount", 10000000000, "Deposit amount in Gwei")
+	depositCmd.Flags().Uint64("key-index", 0, "Internal builder key index to deposit for (0 = the entry key)")
 	depositCmd.Flags().Bool("wait", true, "Wait for deposit to be included")
 	depositCmd.Flags().Duration("timeout", 5*time.Minute, "Timeout for waiting")
 }

cmd/exit.go

@@ -10,7 +10,6 @@ import (
 	"github.com/ethpandaops/buildoor/pkg/lifecycle"
 	"github.com/ethpandaops/buildoor/pkg/rpc/beacon"
 	"github.com/ethpandaops/buildoor/pkg/rpc/execution"
-	"github.com/ethpandaops/buildoor/pkg/signer"
 	"github.com/ethpandaops/buildoor/pkg/wallet"
 )
 
@@ -52,13 +51,20 @@ var exitCmd = &cobra.Command{
 		}
 		defer rpcClient.Close()
 
-		// Initialize BLS signer (raw hex key or mnemonic-derived)
-		blsSigner, err := signer.NewBuilderSigner(cfg.BuilderPrivkey, cfg.BuilderMnemonic, cfg.BuilderKeyIndex)
+		// Initialize the managed builder key set and select the key to exit
+		registry, err := newKeyRegistry(cfg, logger)
 		if err != nil {
-			return fmt.Errorf("invalid builder key: %w", err)
+			return err
 		}
 
-		pubkey := blsSigner.PublicKey()
+		keyIndex, _ := cmd.Flags().GetUint64("key-index")
+
+		key, err := registry.Key(keyIndex)
+		if err != nil {
+			return err
+		}
+
+		pubkey := key.Pubkey()
 
 		// Initialize wallet (its address is the exit source; must match the builder's
 		// registered execution address)
@@ -97,12 +103,13 @@ var exitCmd = &cobra.Command{
 		}
 
 		logger.WithFields(map[string]any{
+			"key":           key.String(),
 			"builder_index": builderInfo.Index,
 			"pubkey":        fmt.Sprintf("%x", pubkey[:8]),
 		}).Info("Submitting builder exit")
 
-		exitSvc := lifecycle.NewExitService(chainSvc, blsSigner, w, logger)
-		if err := exitSvc.CreateExit(ctx); err != nil {
+		exitSvc := lifecycle.NewExitService(chainSvc, w, logger)
+		if err := exitSvc.CreateExit(ctx, key); err != nil {
 			return fmt.Errorf("failed to submit builder exit: %w", err)
 		}
 
@@ -114,4 +121,6 @@ var exitCmd = &cobra.Command{
 
 func init() {
 	rootCmd.AddCommand(exitCmd)
+
+	exitCmd.Flags().Uint64("key-index", 0, "Internal builder key index to exit (0 = the entry key)")
 }

cmd/keys.go

@@ -0,0 +1,28 @@
+package cmd
+
+import (
+	"fmt"
+
+	"github.com/sirupsen/logrus"
+
+	"github.com/ethpandaops/buildoor/pkg/builder_keys"
+	"github.com/ethpandaops/buildoor/pkg/config"
+	"github.com/ethpandaops/buildoor/pkg/signer"
+)
+
+// newKeyRegistry builds the managed builder key set from the configured entry
+// key (raw private key or mnemonic + account index). Internal key 0 is the entry
+// key itself, so a single-key deployment keeps its identity.
+func newKeyRegistry(cfg *config.Config, log logrus.FieldLogger) (*builder_keys.Registry, error) {
+	entryPrivkey, err := signer.ResolveEntryPrivkey(cfg.BuilderPrivkey, cfg.BuilderMnemonic, cfg.BuilderKeyIndex)
+	if err != nil {
+		return nil, fmt.Errorf("invalid builder key: %w", err)
+	}
+
+	registry, err := builder_keys.NewRegistry(cfg, entryPrivkey, log)
+	if err != nil {
+		return nil, fmt.Errorf("failed to initialize builder key registry: %w", err)
+	}
+
+	return registry, nil
+}

cmd/root.go

@@ -65,6 +65,11 @@ func init() {
 	rootCmd.PersistentFlags().Bool("builder-api-on-demand-build", defaults.BuilderAPI.OnDemandBuild, "Build a payload on the fly when a bid request asks for a legal parent no candidate covers yet")
 	rootCmd.PersistentFlags().String("builder-api-url", defaults.BuilderAPI.BuilderURL, "Publicly reachable URL of this builder (e.g. https://builder.example.com); used to validate builder_url in SignedRequestAuthV1")
 	rootCmd.PersistentFlags().Bool("builder-api-require-auth", defaults.BuilderAPI.RequireRequestAuth, "Require SignedRequestAuthV1 on getExecutionPayloadBid requests; reject unauthenticated requests with 401")
+	rootCmd.PersistentFlags().Uint64("builder-keys-target", defaults.BuilderKeys.TargetCount, "Number of builder keys to keep registered and funded (derived from the entry key; index 0 is the entry key itself)")
+	rootCmd.PersistentFlags().Uint64("builder-keys-max-index", defaults.BuilderKeys.MaxIndex, "Highest internal builder key index that may be derived")
+	rootCmd.PersistentFlags().Uint64("builder-keys-discovery-gap", defaults.BuilderKeys.DiscoveryGap, "Number of consecutive unused indices that ends the startup scan for previously deposited keys")
+	rootCmd.PersistentFlags().Bool("builder-keys-auto-deposit", defaults.BuilderKeys.AutoDeposit, "Deposit new builder keys to reach the target count")
+	rootCmd.PersistentFlags().Bool("builder-keys-auto-exit", defaults.BuilderKeys.AutoExit, "Exit surplus builder keys when the managed count exceeds the target (irreversible)")
 	rootCmd.PersistentFlags().Uint64("deposit-amount", defaults.DepositAmount, "Builder deposit amount in Gwei")
 	rootCmd.PersistentFlags().Uint64("topup-threshold", defaults.TopupThreshold, "Balance threshold for auto top-up in Gwei")
 	rootCmd.PersistentFlags().Uint64("topup-amount", defaults.TopupAmount, "Amount to top-up in Gwei")
@@ -92,6 +97,10 @@ func init() {
 	rootCmd.PersistentFlags().Uint64("epbs-bid-value-override", defaults.EPBS.BidValueOverride, "Absolute p2p bid base value in gwei, replacing max(blockValue, bid-min) + subsidy (0 = disabled); allows underbidding the block value for testing")
 	rootCmd.PersistentFlags().Uint64("epbs-vote-threshold", defaults.EPBS.HeadVoteThresholdPct, "Head-vote participation threshold in percent; crossing it fires an immediate threshold_met update (0 = disabled)")
 	rootCmd.PersistentFlags().String("epbs-bid-candidate", defaults.EPBS.BidCandidate, "Which built candidate payload p2p bids commit to: auto, parent_full, parent_empty, grandparent_full, grandparent_empty or all")
+	rootCmd.PersistentFlags().String("epbs-key-strategy", defaults.EPBS.KeyStrategy, "Which managed builder key signs each bid: round_robin, single, random or least_used")
+	rootCmd.PersistentFlags().Uint64("epbs-bid-keys-per-slot", defaults.EPBS.BidKeysPerSlot, "Max distinct builder keys bidding one slot (0 = no cap beyond the fleet)")
+	rootCmd.PersistentFlags().Uint64("epbs-bid-keys-per-step", defaults.EPBS.BidKeysPerStep, "Builder keys bidding a payload per interval step, each one increment higher (0 = every remaining key at once)")
+	rootCmd.PersistentFlags().String("builder-api-key-strategy", defaults.BuilderAPI.KeyStrategy, "Which managed builder key signs served Builder API bids (empty = follow --epbs-key-strategy)")
 	rootCmd.PersistentFlags().Bool("epbs-bid-candidate-switch", defaults.EPBS.BidCandidateSwitch, "Allow the auto bid candidate selection to switch mid-slot when the chain view changes")
 
 	// Payload build candidates (reorg / payload-miss preparedness)
@@ -199,6 +208,14 @@ func initConfig() error {
 			ValueOverrideGwei:     v.GetUint64("builder-api-value-override"),
 			ServeCandidates:       v.GetString("builder-api-serve-candidates"),
 			OnDemandBuild:         v.GetBool("builder-api-on-demand-build"),
+			KeyStrategy:           v.GetString("builder-api-key-strategy"),
+		},
+		BuilderKeys: config.BuilderKeysConfig{
+			TargetCount:  v.GetUint64("builder-keys-target"),
+			MaxIndex:     v.GetUint64("builder-keys-max-index"),
+			DiscoveryGap: v.GetUint64("builder-keys-discovery-gap"),
+			AutoDeposit:  v.GetBool("builder-keys-auto-deposit"),
+			AutoExit:     v.GetBool("builder-keys-auto-exit"),
 		},
 		DepositMaxFeeGwei: v.GetUint64("deposit-max-fee"),
 		DepositAmount:     v.GetUint64("deposit-amount"),
@@ -223,6 +240,9 @@ func initConfig() error {
 			HeadVoteThresholdPct: v.GetUint64("epbs-vote-threshold"),
 			BidCandidate:         v.GetString("epbs-bid-candidate"),
 			BidCandidateSwitch:   v.GetBool("epbs-bid-candidate-switch"),
+			KeyStrategy:          v.GetString("epbs-key-strategy"),
+			BidKeysPerSlot:       v.GetUint64("epbs-bid-keys-per-slot"),
+			BidKeysPerStep:       v.GetUint64("epbs-bid-keys-per-step"),
 		},
 		Reveal: config.RevealConfig{
 			Enabled:             v.GetBool("reveal-enabled"),