henyey mainnet daily — 2026-05-25

[tomerweller]

A more active day than the last several — a 4-commit deploy went out at 17:17 UTC yesterday (6011ec9c → 1a3b07106c3f, bundling HERDER §5.2 observer txset parity #2875, OVERLAY/LEDGER spec refresh #2884, and two skill/doc commits). The new binary survived its first #2791 INV-H2 self-correct cycle this morning (07:44 UTC) — the corrective branch fired exactly as designed when LCL caught up to tracking after a 7-flip V→CU→V cluster.

Validator

• Build: 1a3b07106c3f (uptime 20h 14m; 1 deploy in last 24h, ~5m 30s build time)
• State: 🟢 Validating (age=4s, agree=21/missing=0, lag_ms=444)
• SCP propagation (first_to_self_externalize_seconds): 167 ms (current scalar)
• SCP slot-cycle (externalized_seconds): 5.02 s
• Memory: RSS 16.2 G, frag 21.5 % (heap_components_mb 5692 → 5698 → 5702 over last three reports — flat; RSS down ~4 G from yesterday's 20.0 G because the process is post-deploy fresh)
• Disk: data 84 % (1.4 T of 1.7 T), session 3.1 G, mainnet 47 G (mainnet grew ~6 G in 24h — normal close churn)

Deploys (1)

• 1a3b07106c3f Bundled deploy at 17:17 UTC 2026-05-24 — checked out tip of origin/main covering 4 commits:
  • 1a3b0710 Handle deferred auto-merge in /review-pr skill (Handle deferred auto-merge in /review-pr skill #2885) — .github + scripts
  • 9b6a0464 Refresh OVERLAY and LEDGER spec-adherence reports (Refresh OVERLAY and LEDGER spec-adherence reports #2884) — crates/ledger + crates/overlay
  • 29b70697 Implement observer txset build + cache parity (HERDER §5.2) (Implement observer txset build + cache parity (HERDER §5.2) #2875) — crates/app + crates/herder
  • 0aa3599b Add cycle-1 complete-list discipline and force-converge at lifetime cap — .github
  • Watch sentinel: fired and recovered — first INV-H2 self-correct on the new binary at 07:44 UTC cleared cleanly via INV-H2 panic at herder.rs:940 during catchup replay (separate from #2789) #2791 corrective branch (lcl=tracking=62726271, gap=1); validator stayed up throughout

77c09eee Make agent-worktree-contract canonicalization portable (scripts-only) landed mid-window but did not trigger rebuild.

Incidents

(none) — no crashed-* / frozen-* rotations, no urgent issues filed, no MONITOR ACTION ticks. Single preredeploy rotation (monitor.log.preredeploy-20260524T171737Z) at the planned deploy.

Issues activity

Filed today (6):

• ⚪ Force-converge residuals from plan #2820 (HERDER §5.3 + §15.3 round-2 REVISE-MAJOR) #2895 Force-converge residuals from plan HERDER spec adherence: Finish remaining persistence and recovery partial audit rows #2820 (HERDER §5.3 + §15.3 round-2 REVISE-MAJOR)
• ⚪ Force-converge residuals from PR #2885 (issue #2877) #2894 Force-converge residuals from PR Handle deferred auto-merge in /review-pr skill #2885 (issue Pipeline merge permission gap for PR #2875 #2877)
• ⚪ Force-converge residuals from PR #2884 (issue #2873) #2893 Force-converge residuals from PR Refresh OVERLAY and LEDGER spec-adherence reports #2884 (issue Refresh OVERLAY + LEDGER spec-adherence audits against current main #2873)
• ⚪ Force-converge residuals from PR #2875 (issue #2869) #2892 Force-converge residuals from PR Implement observer txset build + cache parity (HERDER §5.2) #2875 (issue HERDER §5.2: Non-validator txset build + cache parity #2869)
• ⚪ Force-converge residuals from PR #2865 (issue #2837) #2891 Force-converge residuals from PR Enforce $HOME/data validation for reviewer/critic workspace overrides #2865 (issue Enforce $HOME/data validation for reviewer/critic workspace overrides #2837) (closed)
• ⚪ Force-converge residuals from PR #2835 (issue #2816) #2890 Force-converge residuals from PR Implement ctValidityOffset trigger-time and far-ahead abort semantics #2835 (issue HERDER spec adherence: Align ctValidityOffset trigger-time and abort semantics #2816)

All 6 are force-converge follow-ups (the round-2 cap mechanism mentioned in #2870/0aa3599b just kicked in for the first time — the planner is filing residual REVISE-MAJOR concerns instead of blocking).

Closed today (4):

• Force-converge residuals from PR #2865 (issue #2837) #2891 Force-converge residuals from PR Enforce $HOME/data validation for reviewer/critic workspace overrides #2865 (filed and closed same day)
• Pipeline merge permission gap for PR #2875 #2877 Pipeline merge permission gap for PR Implement observer txset build + cache parity (HERDER §5.2) #2875 (resolved by the Handle deferred auto-merge in /review-pr skill #2885 merge)
• Refresh OVERLAY + LEDGER spec-adherence audits against current main #2873 Refresh OVERLAY + LEDGER spec-adherence audits (resolved by 9b6a0464)
• HERDER §5.2: Non-validator txset build + cache parity #2869 HERDER §5.2: Non-validator txset build + cache parity (resolved by 29b70697)

Still open (selected, sorted by last activity):

• ⚪ HERDER spec adherence: Finish remaining persistence and recovery partial audit rows #2820 HERDER spec adherence: Finish remaining persistence and recovery partial audit rows — updated 2026-05-25 (active in /plan)
• ⚪ Enforce $HOME/data validation for reviewer/critic workspace overrides #2837 Enforce $HOME/data validation for reviewer/critic workspace overrides — updated 2026-05-24
• ⚪ HERDER spec adherence: Align ctValidityOffset trigger-time and abort semantics #2816 HERDER spec adherence: Align ctValidityOffset trigger-time and abort semantics — updated 2026-05-24
• ⚪ /review-pr sub-agents create huge worktrees in /tmp (11GB seen, near-OOM root FS) #2843 /review-pr sub-agents create huge worktrees in /tmp (11GB seen, near-OOM root FS) — updated 2026-05-21
• ⚪ Spec adherence: LEDGER — implement INV-L4 ConservationOfLumens #2804 Spec adherence: LEDGER — 18 findings (79%) — updated 2026-05-20
• ⚪ HERDER: Wire ScpPersistenceManager::restore_scp_state into startup #2769 HERDER: Wire ScpPersistenceManager::restore_scp_state into startup — updated 2026-05-20
• ⚪ Spec adherence: HERDER — 10 findings (86%) #2803 Spec adherence: HERDER — 10 findings (86%) — updated 2026-05-20
• ⚪ Spec adherence audit: suite overview (apply mode, v26.0.1 canonical) #2808 Spec adherence audit: suite overview (apply mode, v26.0.1 canonical) — updated 2026-05-19
• ⚪ Spec adherence: OVERLAY — 19 findings (71%) #2802 Spec adherence: OVERLAY — 19 findings (71%) — updated 2026-05-19
• ⚪ HERDER: Event-driven consensus trigger scheduling (Phase 2) #2702 HERDER: Event-driven consensus trigger scheduling (Phase 2) — updated 2026-05-18

Watch items

• INV-H2 WARN count: 1 in last 24h — the first since the post-recovery startup. Fired at 07:44 UTC on the new 1a3b07106c3f binary, concluding a 7-flip V→CU→V cluster (07:34–07:44). LCL=tracking=62726271, gap=1, self-corrected per INV-H2 panic at herder.rs:940 during catchup replay (separate from #2789) #2791 design. Worth tracking the per-binary rate to see if the HERDER §5.2 change (Implement observer txset build + cache parity (HERDER §5.2) #2875) affects how often this scenario arises.
• State transition clusters: 30 transitions in the 24h window, dominated by three flip clusters — yesterday's 16:11–16:15 cluster (4 flips, on the old 6011ec9c binary), the post-deploy initialization at 17:17–17:19 (4 transitions, expected), the 18:25–18:30 cluster (12 transitions on the new binary), and today's 07:34–07:44 cluster (14 transitions ending with the INV-H2 self-correct). Pattern persists across the binary swap.
• Disk usage: surged 76 → 87 % mid-day during deploy + agent activity, then dropped back to 79 % via an external cleanup (~67 G freed in ~26 min around 19:35–20:01 UTC). Now steady at 84 % / 279 G free.
• RSS post-deploy: started at ~9.5 G, climbed to 16.2 G over 20h (~340 MB/h average — matches the post-restart settling pattern). Heap-components flat at ~5.7 G; growth in anon_rss_mb (now 14.7 G) and mmap_mb (14.8 G).
• Daily-summary cron: missed firing today at 13:07 (25 min past at post time). This is the 4th miss in the last 7 days. The cron's reliability is degrading further — overdue to move to a host-side scheduler.

Tick aggregates (last 24h)

• Cron: 6c74937d (daily-summary, recurring 13:07 UTC) — missed today; this is the 4th miss in 7 days. Monitor-tick remains on a ScheduleWakeup dynamic loop (~30 m cadence).
• Ticks observed: 48 (close to nominal 48 at 30 m cadence; includes 1 short-cadence tick at 25 min after the bash-tool outage at 23:28 UTC yesterday)
• Outcomes: 46 OK / 1 DEPLOY / 1 DEPLOY-PENDING / 0 ACTION / 0 WARNING / 0 OFFLINE
• Alarm-firings: 0
• Self-reflection events: 0
• Skill commits today: 0 (the .github/skills/ commits — 0aa3599b, 1a3b0710 — happen to touch the GitHub-side skill definitions, not .claude/skills/)

Open questions

• The 07:44 UTC INV-H2 self-correct was the first on the new binary that includes HERDER §5.2 (29b70697 observer txset build + cache parity). Worth establishing a baseline rate per binary — is one self-correct per ~14 hours expected, or is the new binary triggering them more/less often than 6011ec9c did? The session-total INV-H2 count is the best signal we have today.
• Harness Bash tool returned exit 1 on the 2026-05-24 23:28 UTC tick (recovered by 23:39). Validator wasn't affected, but worth noting as a monitor-side reliability event — should we capture this to the watch array as a harness_outage key for cross-tick aggregation?
• Daily-summary cron has now missed 4 of the last 7 scheduled fires. The manual fallback is reliable but expensive per day. Time to move the post to a host-side scheduler.