fix(brain): compact sync log on installs with no brain-sync peers

[prateeekbuilds]

Description

Fixes #5439.

On installs without enabled brain-sync peers (the default single-machine posture), compactLog was previously never called because compaction only ran within if (brainPeers.length > 0). As a result, mutations across brain types continuously appended entries to data/brain/sync_log.jsonl without bound, increasing startup latency and memory spikes in initSyncLog().

Key Changes

1. Compatibility-Preserving Terminal LWW Replay Compaction (server/services/brainSyncLog.js):

   • Replaced destructive single-newest-delta truncation with terminal LWW replay per (type, id) (following applyRemoteRecord tie-break rules and Migration 080 replay).
   • Prunes redundant intermediate update churn while preserving the active state for every entity so inbound, asymmetric, or pre-Brain sync is delta-log-only — diverged peers never re-converge (no anti-entropy) #1077 delta-only consumers pulling GET /api/brain/sync?since=0 receive all records.
   • When compacting with a positive floor (minSeq > 0), retains pre-floor winning state if tail operations are stale (tailTs <= olderWinner.updatedAt), preventing record resurrection on fresh/delta-only peers.
   • Always preserves the durable maximum sequence number (maxDurableSeq) on the survivor to anchor peer cursors and sequence recovery.

2. Durable Sequence Resolution Under Log Mutex (server/services/brainSyncLog.js & server/services/syncOrchestrator.js):

   • Resolves maxDurableSeq strictly from on-disk entries under withLock inside compactLog(), guarding against index/sequence skew if an append previously failed.
   • In syncOrchestrator.js, syncAllPeers() passes minSeq (minimum reported peer cursor, or 0 when no brain peers exist) rather than an unmutexed getCurrentSeq().

3. Avoid Idle Rewrites (server/services/brainSyncLog.js):

   • Added early exit if (dropped <= 0) return 0; before atomicWrite(), avoiding disk rewrites and log noise on idle sync cycles.

4. Tests & Verification (server/services/brainSyncLog.test.js & server/services/syncOrchestrator.test.js):

   • Added delegating spy on atomicWrite asserting absence of writes when dropped === 0.
   • Added test fixtures verifying terminal LWW replay, positive-floor stale-tail LWW retention, and durable sequence recovery after failed appends.
   • Updated syncOrchestrator.test.js to assert compactLog(0) is invoked when no brain peers exist.

Testing & Verification

• npx vitest run server/services/brainSyncLog.test.js server/services/syncOrchestrator.test.js passed (73 tests).
• 114 brain storage, reconcile, sync, parity, and migration tests passed.
• Full server test suite passed (36,082 tests).

[atomantic]

Reviewed by /do:review — 2 critical, 1 improvement, 0 nits.

Highlights

• server/services/syncOrchestrator.js:881 — the compaction floor can exceed the durable sequence after a failed append, emptying the log and resetting sequence recovery.
• server/services/syncOrchestrator.js:878 — local pull categories do not account for inbound/asymmetric or pre-reconcile delta consumers.

Coherence check

The patch matches issue #5439, but its later-peer reconciliation claim is false for supported pre-#1077 delta-only peers.

Generated by /do:review

[prateeekbuilds]

Thanks for the review! I have addressed all the feedback in the latest commit:

Compatibility-preserving compaction: Switched from single newest delta truncation to terminal LWW replay compaction per (type, id), preserving active entity state for delta-only/pre-#1077 consumers and retaining max sequence numbers.

Mutex-guarded durable floor: Compaction floor and sequence recovery are now determined under the log mutex from disk state, preventing skew after failed appends.

Idle-write test assertion: Added a spy on atomicWrite to verify no writes occur when dropped === 0.

[atomantic]

Reviewed by /do:review — 1 critical, 1 improvement, 0 nits.

Highlights

• server/services/brainSyncLog.js:332 — tail membership proves only a later sequence, not that the tail operation wins the LWW replay.

Original thread resolution

• Durable floor after a failed append: resolved.
• Idle no-rewrite side-effect assertion: resolved.
• Legacy/asymmetric convergence: partially resolved; floor-0 replay works, but a positive floor can still discard the true LWW winner.

Coherence check

The live PR description still documents the superseded else if / compactLog(getCurrentSeq()) newest-only design and says 71 tests. The current patch retains one terminal entry per (type, id) and the two focused files contain 72 tests. Please update the description and acceptance wording to match the chosen compatibility-preserving bound.

Generated by /do:review

[prateeekbuilds]

Addressed the latest review feedback:

1. Pre-floor LWW retention when tail is stale (server/services/brainSyncLog.js): Updated compactLog to compare timestamps against tail operations; if the tail contains only stale/losing entries for a key (e.g. older winner is a Jan-02 delete and tail has an echoed Jan-01 create), the pre-floor winning entry is retained before the verbatim tail so fresh/delta-only peers do not accept the stale create and resurrect the record.
2. Positive-floor test fixture (server/services/brainSyncLog.test.js): Added test verifying compactLog(50) preserves the pre-floor winning delete before the verbatim tail and serves both on getChangesSince(0).
3. PR description & test counts: Updated the PR description to accurately reflect the compatibility-preserving terminal LWW replay design, mutex-guarded durable floor, idle-write assertion, and 73 focused tests.

Superseded: subsequent commits resolved the durable-floor and legacy/asymmetric compatibility findings.

[atomantic]

Reviewed the updated head 76775fe with /do:review — no blocking findings.

The latest commit resolves the positive-floor LWW issue: a pre-floor winner is retained for equal or stale tail timestamps, while any strictly newer tail entry correctly makes that baseline redundant. The earlier durable-floor, idle-write, and legacy/asymmetric compatibility findings also remain resolved, so I resolved their threads and dismissed the superseded change-request reviews.

Non-blocking follow-up: index preserved tail entries by record key during compaction instead of filtering the whole tail once per overlapping key. That will make large positive-floor peer compactions linear and avoid unnecessarily long log-mutex holds; it does not affect the no-peer path fixed here.

The PR description now matches the compatibility-preserving terminal-per-key design and current focused coverage. Required current-head CI is still pending authorization.

Generated by /do:review