Block `bonds` Field Never Catches Up to `post_state_bonds` After a New Bond Crosses an Epoch Boundary

[9Neechan]

What did you expect to see?

When a new validator's bond deploy is included in a block and that block is finalized, the validator's stake should show up in the bonds map returned by /api/last-finalized-block within one epoch boundary (pos_epoch_length blocks) of the deploy landing — matching what post_state_bonds already reflects at inclusion time.

What did you see instead?

The bond deploy executed successfully on every node (("Bond result:", true, "Message:", Nil) — logged independently on validator1-0, validator2-0, validator3-0, validator4-0, bootstrap-0, and observer-0) and was selected into block 73 (Deploy selection for block 73: ... valid=1, selected=1), which was subsequently finalized. Yet the harness's /api/last-finalized-block poll (pkg/monitor/monitor.go:WaitForBond, reads bestLFB.Bonds) never saw the new validator's pubkey appear, and the smoke experiment aborted after the full 5-minute timeout:

bond for 04b461c2cf47d23f… not present in LFB bonds map after 5m0s (deploy likely never landed)

This is despite the chain staying healthy the entire time — finalized_block climbed monotonically from 47 to 94 over the wait window (smoke-chain.csv), with no freeze, no replay errors, no DAG-merge corruption. That rules out this being another instance of the previously-reported bonding failure classes #51, #104 — both of those involve the network stalling or state corrupting. Here the network kept producing and finalizing blocks fine; only the reported bonds set stayed stale.

Steps to reproduce the bug

1. Start a 3-genesis-validator network with heartbeat enabled, pos_epoch_length=10, pos_quarantine_length=10.
2. Wait for a pre-bond block milestone (70 finalized blocks in this run).
3. Submit a bond deploy for a 4th, already-synced validator.
4. Poll /api/last-finalized-block's bonds field for the new validator's pubkey for at least 5 minutes (well past the next epoch boundary after the deploy lands).

Supporting data (from this run)

• Deploy 3045022100df3dae9b6e... submitted at 10:29:37.448 (validAfterBlockNumber: 70).
• Selected and included in block 73 (seqNum 69) at 10:29:37.822 (alreadyInScope=0, selected=1) — validator1-0.log:5107.
• Bond result: true logged on every node between 10:29:37 and 10:29:54 — validator1-0.log:5116, plus matching lines in bootstrap-0.log, observer-0.log, validator2-0.log, validator3-0.log, validator4-0.log.
• Immediately after, at 10:29:38.631:

  "block bonds field differs from post-state bonds", floor_number: 70, committee: 3, post_state_bonds: 4
  

  — validator1-0.log:5119 (f1r3fly.casper.bonds_validation, block_creator.rs:3098). This confirms the discrepancy exists from the moment the bond lands: post-state already has 4 bonds, but the block's own bonds field (committee) is still 3, carried over from floor_number 70.
• Subsequent attempts to re-include the same deploy in later blocks (75 and beyond) are correctly rejected as duplicates (Deploy ... FILTERED (already in scope), DagMerger rejected 1 user deploys) — validator1-0.log:5177, :5182. This is expected dedup behavior, not the bug; it just confirms the deploy was never re-selected because it was already considered "in scope" from block 73.
• By the time the experiment aborted, finalized_block had reached 94 — several epoch boundaries (80, 90) past block 73 — with no sign of the bonds field ever picking up the 4th validator.
• validator4's PROPOSED count stayed at 0 for the entire run (smoke.txt VALIDATOR BLOCKS table), consistent with it never being recognized as part of the committee.

Root cause hypothesis

The block header's bonds field (what /api/last-finalized-block exposes, and what determines committee membership) is meant to be refreshed from post_state_bonds at each epoch checkpoint (pos_epoch_length=10 blocks). That refresh appears to not be happening — or is happening against the wrong reference point — so a bond that lands mid-epoch never gets picked up by any later block's bonds field, even multiple epoch boundaries later. This looks like a regression in the epoch/committee-update logic specifically, not in bond-deploy execution (which clearly succeeds — Bond result: true everywhere) or in general consensus health (chain kept finalizing normally throughout).

Affected source locations (candidates)

casper/src/rust/blocks/proposer/block_creator.rs:3098   (f1r3fly.casper.bonds_validation — where the discrepancy is logged)

Whatever code path is supposed to promote post_state_bonds into a block's bonds field at the next epoch boundary.

Suggested fix / next steps

• Reproduce with the raw node logs from this run (logs/dev-v/smoke-27-07-26-101109/logs/smoke/{validator1-0,bootstrap-0,observer-0,validator2-0,validator3-0,validator4-0}.log, all around 10:29:37–10:29:55) and step through the epoch-checkpoint code that's supposed to consume post_state_bonds into the block's bonds field.
• Confirm whether the bonds field ever updates at all after enough further epochs elapse (this run only waited to block 94 — 2 epochs past the deploy — before the harness's 5-minute timeout forced an abort), or whether it's permanently stuck.
• Done: widened the smoke/consensus bond-verification wait from 5 to 20 minutes (bondVerifyTimeout in experiment/consensus/runner.go), matching the existing budget already used for the later "Propose started" activation wait.
  • Resolved — permanent staleness confirmed, not a timing issue. Re-ran smoke with the 20-minute timeout: still ABORTED — bond verification validator4: bond for 04b461c2cf47d23f… not present in LFB bonds map after 20m0s, with validator4 PROPOSED=0 for the entire 169-block run (chain healthy and finalizing the whole time, same as the first run). 20 minutes at this run's ~12.4s/block cadence is roughly 9 epoch boundaries past the deploy landing — the bonds field never catches up no matter how much epoch-aligned time it's given. This rules out "the test was just impatient" and confirms the root cause is a genuine node-side regression in promoting post_state_bonds into the block/LFB bonds field. This is now ready to file upstream as-is.

Related issues

• Different failure mode, same general area (bonding a new validator): Multiple validators bonding issue #51 — DAG-merge/state-root corruption, not observed here.
• Different failure mode, same general area: Purse-Deposit Double-Produce Corrupts Number Channel, Halts Entire Network #104 — full network halt, not observed here (chain kept finalizing throughout this run).

Relates tests

smoke-27-07-26-101109
smoke-27-07-26-113545