Clone sync monitor is vacuous on RocksDB: clone marks itself Available mid-copy (targets all 0) #655

Summary

clone_node's sync monitor is vacuous on RocksDB: it declares "All databases synchronized" on its first poll — seconds into a multi-GB base copy — and the clone marks itself availability: Available and cloned with an arbitrarily small fraction of the leader's data. This is long-standing (not a regression from the #649 monitor rework); it reproduces deterministically locally.

Evidence

CI: Large-Data stress run 31006315068 failed Clone row count 31393 != leader 104858; the clone log shows "All databases synchronized" 2.2s after requesting a full copy of a 10 GB database. Green runs (30982928212, 30792466356) show the identical instant-synchronized — they passed only because the copy happened to finish inside the test's row-count polling window. Aug 3's run declared synced 59s into a still-running copy.

Local repro (1 GB, HARPER_RUN_STRESS_TESTS=1 HARPER_STRESS_LARGE_DATA_GB=1 … largeClone.test.mjs) fails identically, and with instrumentation shows the exact decision:

[cloneNode]: [KR-WM] targets={"system":0,"data":0} sockets=[{"db":"system","v":1785939110547},{"db":"data","v":0}]
[cloneNode]: [KR-WM] Database system: No target timestamp, skipping sync check
[cloneNode]: [KR-WM] Database data: No target timestamp, skipping sync check
[cloneNode]: All databases synchronized        <-- first poll, data copy just started

Root cause chain

1. Describe reports no last_updated_record on RocksDB. cloneNode.getLastUpdatedRecord() derives each database's sync target from describe_database/describe_all → last_updated_record. Core's schemaDescribe.ts computes that from auditStore.getKeys({ reverse: true, limit: 1 }) — but RocksTransactionLogStore.getKeys() is an unimplemented stub (return []; // TODO: implement this), and the indices.__updatedtime__ fallback doesn't exist on these tables. So every database's target is 0.
2. The monitor treats a falsy target as "skip this database". checkSyncStatus (cloneNode/syncMonitor.ts) does if (!targetTime) continue; — with every database skipped, syncComplete stays true and the first poll succeeds with zero verification.

The receive-side copy machinery is doing its part correctly: the per-database received-version watermark (RECEIVED_VERSION_POSITION) is deliberately held at 0 for the whole bulk copy and advanced to copyStartTime only by the single end_txn the sender always emits after the copy — the code calls this "the sole signal that the copy is synced". The monitor just never looks at it when targets are missing.

Impact

• Every RocksDB clone completes as "successful" without any sync verification; the node advertises Available to the control plane while the copy is still streaming. Reads served meanwhile silently miss data.
• The stall detection added in Clone sync monitoring times out only when replication stalls, so large clones no longer fail at a fixed 5-minute cap #649 is unreachable (first check always passes), so a genuinely wedged copy is also not caught.
• The Large-Data clone stress test races the copy against its bounded row-count wait: red or green depends on runner I/O throughput.

Fix (this issue → harper-pro)

In checkSyncStatus, a database without a target timestamp must not be skipped: require its received-version watermark to be positive (receivedVersion >= (targetTime || 1)). The watermark can only become positive via the sender's final copy end_txn (or later live traffic), so this keys completion to the copy's own completion signal — correct against any leader version, including leaders whose describe cannot report last_updated_record, and for empty databases (their copy still emits the final end_txn). Additionally, a database present in the targets but missing from database_sockets must hold syncComplete false, so a not-yet-registered socket can't produce a vacuous pass either.

Follow-up (separate issue → core)

last_updated_record is silently absent from describe_table/describe_all on RocksDB — a user-visible describe regression vs LMDB — because RocksTransactionLogStore.getKeys() is a TODO stub and the transaction log has no tail-read API to implement it cheaply (rocksdb-js TransactionLog exposes only forward query(); last-committed position exists but carries no timestamp). Needs a small rocksdb-js tail API (e.g. getLastEntry()), then core can restore the field.