Mid-copy kill can resume via the direct seq cursor instead of the copy cursor, permanently skipping uncopied rows

Summary

A clone killed mid-bulk-copy can reconnect via the direct seq-cursor path instead of resuming the copy, permanently skipping the not-yet-copied rows. Observed live in CI (Integration 3/3 on PR #657, cloneResume.test.mjs); rare — the same test passes locally and usually in CI.

Evidence (from the failing run's log dump)

First clone attempt (killed at 15:45:25):

system's copy completed 15:44:28 (persisted shared-status watermark 1785944668706.3).
data's copy was still in flight: its received-version watermark was 0 at the kill (shared status survives restart on RocksDB — the second process still shows first-attempt-era stamps for system).

Restart at 15:45:26:

Connected to ws://127.0.0.3:9933, db: data
469-1c:9933 1 armed leading-duplicate fast-skip 127.0.0.3 1 data direct resume cursor 1785944669174

No Requesting full copy of database data — compare a healthy resume (local run of the same test), where the restart logs Requesting full copy of database data and completes. 1785944669174 ≈ 15:44:29.17, the copy-start era — a seq cursor persisted mid-copy.

Result: the leader replays nothing (the 4000 rows' audit versions predate the cursor), data receives zero frames for the whole 120s window, and the clone is missing most of the table with no pending work anywhere.

Invariant that should hold

While a bulk copy is incomplete, the persisted resume state must never select the direct-cursor path on reconnect: either a copy cursor exists (copy resume) or no cursor exists (fresh full copy). A seq cursor persisted mid-copy (e.g. via seqUpdateEndTxn from a SEQUENCE_ID_UPDATE / empty-txn sequence update — those paths persist [seq] during the copy) combined with an absent/unusable copy cursor (e.g. copyFromNodeId undefined at COPY_START, or a kill before the first COPY_CURSOR_FLUSH_INTERVAL_MS flush) breaks it.

Interaction with #655 / PR #657

Before the sync-monitor fix, this scenario either failed the test's row-count assert or, in production, silently produced a clone missing data while Available. With the monitor fix the clone correctly stays Unavailable (nothing ever arrives, the stall detector eventually fails the clone) — fail-safe, but the underlying skipped-copy state is the real defect. This is the live confirmation of the mid-copy cursor-advance hazard in the SEQUENCE_ID_UPDATE handler (same family as #426/#421).

Repro direction

Needs a targeted repro forcing the window: kill the clone after the first mid-copy sequence-update persists [seq] but before the first copy-cursor flush (REPLICATION_COPYCURSORFLUSHINTERVALMS high + kill early), then restart and observe the direct-cursor reconnect with no copy re-request.

Re-read the evidence: the filed mechanism isn't what that run shows

Going back to the failing run's log for the timing of the data seq write:

15:44:29.353  469-1c:9933 7 bulk copy complete from 127.0.0.3
15:44:29.392  last sequence committed 2026-08-05T15:44:29.174Z data
...60s of silence...
15:45:25.878  Receive watchdog: no activity from 127.0.0.3 (db: "data") for 60000ms
15:45:26.382  Starting time recorded in db 127.0.0.3 1 data 1785944669174 start time: 1785944669174

1785944669174 was committed 39 ms after that database's COPY_COMPLETE, and it is copyStartTime. That is the designed end-of-copy cursor advance (maybeFinishCopy after the final end_txn), not a cursor persisted mid-copy. So the restart's direct-cursor resume was correct behaviour given the state on disk — the state was already wrong before the kill: data reported a completed copy in 180 ms with rows missing, then went silent for 60 s until the watchdog reconnected. The kill only exposed it.

The filed mechanism is not reachable on `main`

Tried to build the repro the issue asks for (2-node add_node, both copy-cursor flush gates pushed out of reach — interval 1 h, bytes 1 GiB — plus a churn writer on the source, SIGKILL mid-copy at 20–70% of the rows, restart on the same data dir). Three runs, and in every one the restarted receiver had no seq row at all:

Starting time recorded in db <leader> 1 data undefined start time: 1 1970-01-01T00:00:00.001Z
Requesting full copy of database data from wss://<leader>:9933

Reason: no sequence updates flow during a base copy. Copy checkpoints deliberately carry none (#255, 2026-06-02 — "a plain flush with NO sequence update"), and copy-frame end_txns use lastSequenceIdReceived, which stays undefined for the whole copy — the receiver logs last sequence committed Invalid Date for every copy batch. Live source writes are not interleaved either: the leader's send loop runs the copy to completion before the audit replay. So {seq > 1, no copyCursor, copy incomplete} did not occur, and the direct-resume path was never selected.

Where that leaves it

The invariant is still unenforced — incompleteCopyForcesFullCopy did not exist; a surviving-but-cursorless interrupted copy has nothing that vetoes the direct path. Reachable in principle when a node that already has a seq baseline is pushed into a base copy (retention-forced resync, relayed lastTxnTime from a proxy peer), though in the retention case the leader's own guard re-forces the copy.
The observed signature belongs elsewhere: a copy that reports COPY_COMPLETE while its rows are missing is the finalization/apply family — fix(replication): guard the base-copy finalization window so a copy that cannot finish is reported and retried #678 (guard the base-copy finalization window) and fix(replication): classify undecodable records + hold on unknown table id (#537) #545 (hold on unknown table id) both merged after 2026-08-05.

Downgrading P0 → P2: keeping the invariant guard as defense-in-depth, but the filed data-loss path is not demonstrable on main and its live signature is covered by fixes that landed since.

Hardening branch: kris/658-copy-incomplete-marker — durable [copyIncomplete, nodeName] marker written at COPY_START, cleared only in maybeFinishCopy's fully-durable block, vetoing the direct-cursor selection; unit coverage for the predicate. No fails-on-base integration test, for the reason above.

fix(replication): scope the marker sweep to cluster membership, not o…

Navigation Menu

Repository navigation

Mid-copy kill can resume via the direct seq cursor instead of the copy cursor, permanently skipping uncopied rows #658

Summary

Evidence (from the failing run's log dump)

Invariant that should hold

Interaction with #655 / PR #657

Repro direction

Re-read the evidence: the filed mechanism isn't what that run shows

The filed mechanism is not reachable on `main`

Where that leaves it

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Mid-copy kill can resume via the direct seq cursor instead of the copy cursor, permanently skipping uncopied rows #658

Description

Summary

Evidence (from the failing run's log dump)

Invariant that should hold

Interaction with #655 / PR #657

Repro direction

Activity

kriszyp commented on Aug 20, 2026

Re-read the evidence: the filed mechanism isn't what that run shows

The filed mechanism is not reachable on main

Where that leaves it

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions

The filed mechanism is not reachable on `main`