henyey mainnet daily — 2026-05-19

[tomerweller]

Validator

• Build: 4812a4be (uptime 4h 27m; 3 deploys in last 24h, ~5m avg/build)
• State: 🟢 Validating (agree=21/missing=0, lag_ms=494)
• SCP propagation (first_to_self_externalize_seconds): 65ms
• SCP slot-cycle (externalized_seconds): 5.59s
• Memory: RSS 11.3G, frag 16% (heap_components_mb=5927 stable)
• Disk: data 56% (922G of 1.7T), session 2.6G, mainnet 29G

Deploys (3 in window)

• 4812a4be Instrument soroban_state Arc strong_count for diagnostic (Instrument soroban_state Arc strong_count for diagnostic #2812)
  • Files touched: crates/app, crates/ledger
  • Watch sentinel: silent (currently deployed; clean catchup-replay, 2 INV-H2 corrective WARNs from initial replay only)
• a5153b86 Fix SCP spec adherence: INV-S9 assertion, §8.2 iteration cap, anchors (Fix SCP spec adherence: INV-S9 assertion, §8.2 iteration cap, anchors #2809)
  • Files touched: crates/scp
  • Watch sentinel: silent (clean deploy + ~2h uptime before replaced by Instrument soroban_state Arc strong_count for diagnostic #2812)
• e851f037 Add per-crate SPEC_ADHERENCE.md from /spec-adhere --apply (with Make SCP tx-set purge transactional #2799 SCP tx-set purge + Wire ScpPersistenceManager::persist_scp_state into ScpDriver::emit (#2768) #2795 ScpPersistenceManager wire bundled in same deploy)
  • Files touched: crates/{bucket,herder,history,ledger,overlay,scp,tx}, crates/{db,herder}, crates/herder
  • Watch sentinel: rolled back (process died at 03:22 from /home/tomer/data full disk — see Incidents)

Incidents (1 major resolved)

• 🟢 Disk-full validator crash
  • Window: 2026-05-19T03:22Z → 2026-05-19T05:17Z (1h55m total downtime)
  • Symptom: flush_pending_persist: bucket ...: IO error: No space left on device (os error 28) — validator process exited; /home/tomer/data was 100% (1.7T/1.7T)
  • Root cause: accumulated stale agent worktrees (pdr-, do-, timestamped dated backups, henyey-3 build cache) filled the 1.7T volume; no automatic cleanup
  • Resolved by: operator-approved cleanup of 23 pdr-* + ~25 stale do-* + 14 timestamped + henyey-3 (~1T freed); validator relaunched on e851f037 at 05:16:19 UTC, reached Validating ~11min later after replaying L62633607 → L62634905 (~1300 ledgers, 4 INV-H2 corrective WARNs handled cleanly)

Issues activity

Filed today (26):

• ⚪ Implement §12 + INV-C6 verify_tx_result_set OFFLINE_COMPLETE caller loops #2831 Implement §12 + INV-C6 verify_tx_result_set OFFLINE_COMPLETE caller loops
• ⚪ Plumb §9.1 + INV-C5 TrustSource::Scp through callers #2830 Plumb §9.1 + INV-C5 TrustSource::Scp through callers
• ⚪ Implement §6.1 mode discriminator enum (OFFLINE_BASIC/OFFLINE_COMPLETE/ONLINE) #2829 Implement §6.1 mode discriminator enum
• ⚪ Investigate §14.5 REBUILD_FOR_OFFER_TABLE flag (absent) #2828 Investigate §14.5 REBUILD_FOR_OFFER_TABLE flag (absent)
• ⚪ Reconcile §5.3 publish queue (drift from filesystem spec) #2826/Reconcile §5.3 publish queue (drift from filesystem spec) #2827 Reconcile §5.3 publish queue
• ⚪ LEDGER: Keep soroban_state Arc-count perf fields non-cumulative in LedgerClosePerf aggregation #2823 LEDGER: Keep soroban_state Arc-count perf fields non-cumulative
• ⚪ project-tick-loop: picker should skip items with active foreign sentinels #2822 project-tick-loop: picker should skip items with active foreign sentinels
• ⚪ HERDER spec adherence: Align ctValidityOffset trigger-time and abort semantics #2816-HERDER spec adherence: Finish remaining persistence and recovery partial audit rows #2820 HERDER spec adherence finishing items (5)
• ⚪ LEDGER: Eliminate residual soroban_state COW tail (blocked on #2810 findings) #2811 LEDGER: Eliminate residual soroban_state COW tail (blocked on LEDGER: Instrument soroban_state Arc strong_count to diagnose residual make_mut spikes #2810)
• ⚪ /review-pr reviewer sub-agents skip the structured verdict post #2813//review-pr sub-agents create worktrees in repo root instead of ~/data #2814 /review-pr sub-agent bugs (worktrees in repo root, verdict skip)
• ⚪ Spec adherence: OVERLAY — 19 findings (71%) #2802-Spec adherence audit: suite overview (apply mode, v26.0.1 canonical) #2808 Spec adherence audit rollups (7 area summaries from /spec-adhere)

Closed today (13):

• Align SCP INV-S9 phase guard with stellar-core debug-only assert semantics #2815 66886497 Align SCP INV-S9 phase guard (debug_assert_eq for parity)
• LEDGER: Instrument soroban_state Arc strong_count to diagnose residual make_mut spikes #2810 4812a4be LEDGER: Instrument soroban_state Arc strong_count (Instrument soroban_state Arc strong_count for diagnostic #2812)
• Spec adherence: SCP — 2 findings (96%) #2801 a5153b86 Spec adherence: SCP — 2 findings
• /review-pr SKILL.md PR-detection jq query has bug — closedByPullRequestsReferences has no .state field #2793 b3e4ec83 /review-pr SKILL.md PR-detection jq query
• INV-H2 panic at herder.rs:940 during catchup replay (separate from #2789) #2791 be066514 INV-H2 panic at herder.rs:940 (catchup-replay)
• metrics: mid-run recovery cycle (lost-sync + hard-reset + recovery-stalled) on 52ac1665 at 17h uptime #2789 0df88603 mid-run recovery cycle (hard-reset suppression)
• /review-pr Step 2 not reliably followed by agent — paraphrases rule, ignores SKILL.md update #2787 039c7b5a /review-pr Step 2 not reliably followed
• Follow-up: zsh regression test for monitor-decisions.sh suffix-strip #2782 c33f3b19 zsh regression test for monitor-decisions
• CI: Archive Done Project Items workflow fails — PROJECT_BOARD_TOKEN secret missing or unscoped #2777 74f4b1a2 Archive Done Project Items PROJECT_BOARD_TOKEN
• HERDER: Make ScpPersistenceManager::purge_unreferenced_tx_sets transactional #2770 7380c388 ScpPersistenceManager purge transactional
• HERDER: Wire ScpPersistenceManager::persist_scp_state into envelope-emit hot path #2768 bb431484 ScpPersistenceManager persist wired
• Audit and fix (Owner, TempDir) drop-order pattern across test helpers #2761 22c5a88d (Owner, TempDir) drop-order audit
• SCP persistence: implement hash-aware tx set cleanup #2697 a5153b86 SCP persistence hash-aware tx set cleanup

Still open (long-running):

• enhancement HERDER: Wire ScpPersistenceManager::restore_scp_state into startup #2769 HERDER: Wire ScpPersistenceManager::restore_scp_state into startup
• enhancement HERDER: Event-driven consensus trigger scheduling (Phase 2) #2702 HERDER: Event-driven consensus trigger scheduling (Phase 2)
• ⚪ Spec adherence audit: suite overview (Protocol 26) #2660 Spec adherence audit: suite overview (Protocol 26)

Watch items

• Wedge cycles (metrics: mid-run recovery cycle (lost-sync + hard-reset + recovery-stalled) on 52ac1665 at 17h uptime #2789): 4 wedges observed in last 24h on be06651+e851f037 (all post-Narrow #2713 hard-reset suppression to at-or-near-tip (#2789) #2790 fix). Each ~6m18s, validator survives via INV-H2 panic at herder.rs:940 during catchup replay (separate from #2789) #2791 corrective branch. Inter-wedge intervals 49m-4h12m, highly variable. After 4812a4b deploy: 1 post-deploy wedge cycle then quiet for 4h. Pattern likely ongoing — operator could file follow-up to skip the "archive checkpoint not published yet" gate when peer_gap > N.
• INV-H2 corrective WARNs: cumulative 2 on current build (just from initial post-deploy replay burst). Confined to fast-tracking catchup — not a steady-state concern.
• soroban_state_size: new instrumentation from Instrument soroban_state Arc strong_count for diagnostic #2812 shows ~1.07GB in-memory state; expected baseline. Growth rate not yet characterized over 24h.
• Disk usage: 56% (922G of 1.7T) — growing ~5G/h from active /do worktrees + cargo caches. Periodic cleanup needed. Open question 2 below.

Tick aggregates (last 24h)

• Cron: 6c74937d (daily-summary at 13:07 UTC) — 1 expected, 0 fired today (manual post; same lateness pattern flagged yesterday)
• Outcomes: 34 OK / 1 ACTION / 7 WARNING / 3 DEPLOY (45 ticks; OFFLINE excluded — see disk-full incident in Incidents)
• Alarm-firings: 0 ticks with ≥1 firing alarm (new evaluator-block rows mostly post-recovery; wedge alarms suppressed via cooldown refresh)
• metrics_ratio: 0 breaches, 0 skipped
• recovery_stalled: 0 non-zero-delta ticks (matches: wedges trigger recovery-stalled deltas but all observed in pre-INV-H2 panic at herder.rs:940 during catchup replay (separate from #2789) #2791 era; post-fix wedges register but resolve before crossing streak=3/3 + cooldown filter)
• Skill commits today: 0 (.claude/skills/ untouched)

Open questions

1. Daily-summary cron reliability (continuing from yesterday's henyey mainnet daily — 2026-05-18 #2800 open Q1): 6c74937d ("0 13 * * *") again failed to fire by 13:23. Posted manually for the 2nd consecutive day. Migrating to GH Actions cron is increasingly justified.

2. Disk-full automation: Yesterday's crash (1h55m downtime) was caused by stale agent worktrees filling the 1.7T volume. Worth scripting periodic cleanup of pdr-* (deprecated skill), do-* older than 72h, and timestamped 2026*-* dirs older than 48h? Could be a /monitor-tick check or a separate cron.