Replication W13: Base-copy & catch-up path (consolidation & correctness) #510

Workstream W13 of #430 · base-copy & catch-up path — consolidation & correctness

Summary

Since the epic was filed, the base-copy/catch-up path has become the highest-churn area of replication (the #480 family), but its work is scattered across W2/W3/W10 and one-off fixes. This workstream owns the copy path as a whole: copyApply correctness, checkpointing, live-subscriber visibility, watchdog consolidation, and copy-transport performance.

Shipped baseline (as of 2026-07)

• copyApply durable snapshots — bulk base-copy applied as durable snapshots, not transactions; RocksDB-gated; system DB excluded (Replicated hdb_analytics floods system transaction logs and spins a worker in native code; system base-copy wedges until worker recycle #480, partial)
• Copy-flush pacer under the receive watchdog (fix(replication): pace bulk full-copy flushes under the receive watchdog #483) + backoff on copy-cursor flush failure
• Copy-progress watchdog for ping-alive copy stalls (issue a customer cluster preprod 5.1.7: replicated deploys wedge after rolling upgrade — stalled system blob send (#450/harper#1443) blocks COPY_COMPLETE; fixes absent from 5.1.7 (needs 5.1.8) #453) and a separate copy-phase receive timeout
• Reconcile-level fallback for connected:true/Receiving copy-stall wedges (fix(replication): reconcile-level fallback for connected:true/Receiving copy-stall wedge #463)
• Control-plane-first copy ordering (system base-copy gated on huge hdb_analytics (analytics.replicate:true) blocks hdb_deployment convergence → deploys fail after any copy #421 → Order replication base copy: control-plane tables before bulk tables (#421) #422)
• Cursorless-start full copy (rapid-reconnect stress: follower permanently loses replication backlog after restart churn (data loss; fast-skip ruled out) #426 → fix(replication): full-copy when a source has no resume cursor (#426) #428)

Open pieces

• Replication: txn-log "table-reload" marker to extend copyApply to the system DB (#480 follow-up) #489 — txn-log "table-reload" marker so copyApply can extend to the system DB (Replicated hdb_analytics floods system transaction logs and spins a worker in native code; system base-copy wedges until worker recycle #480 follow-up; Replicated hdb_analytics floods system transaction logs and spins a worker in native code; system base-copy wedges until worker recycle #480 stays open until then)
• MQTT SUBACK granted before cross-node delivery path is live after cluster bootstrap #495 — copyApply rows fire no live subscription events (audit=false), so an already-connected MQTT/SSE/WS subscriber silently misses base-copy rows. Fixes in flight: harper#1530 + harper-pro#507 (stacked on the feat(replication): extend copyApply to the system DB via reload markers #491 / harper#1491 reload-marker primitive)
• Clone bulk table-copy restarts from zero after ping-timeout disconnect (no mid-copy checkpoint) #241 — mid-copy checkpoint: bulk copy restarts from zero after a ping-timeout disconnect
• Direct binary relay for the copy path (from W10) — stop decoding every record only to re-encode it; the biggest copy-CPU win, and it shrinks the system base-copy gated on huge hdb_analytics (analytics.replicate:true) blocks hdb_deployment convergence → deploys fail after any copy #421/NATS MAX_PAYLOAD_EXCEEDED error when cluster cloning large data #266 surface
• Consolidate the copy watchdog/pacer layers onto W1's level-triggered health state once W1 lands — they should demote to telemetry/assertions (see the W1 status update in Replication W1: Connection & health — single source of truth #431)
• Reuse one copy-flush pacer between the fix(replication): pace bulk full-copy flushes under the receive watchdog #483 pacer and the copyApply flush cadence

Dependencies

Rides W1 (watchdog consolidation); feeds W8 (copy progress as a first-class signal); the binary-relay item coordinates with W11's copyTransfer.ts extraction.

Effort / risk

M / medium. The correctness items (#495, #489) are already in flight or well-scoped; the transport perf piece is the larger lift.

Acceptance criteria

• A live subscriber observes base-copy rows (or a re-delivered snapshot) — no silent miss (MQTT SUBACK granted before cross-node delivery path is live after cluster bootstrap #495).
• An interrupted bulk copy resumes from a checkpoint instead of restarting from zero (Clone bulk table-copy restarts from zero after ping-timeout disconnect (no mid-copy checkpoint) #241).
• copyApply covers the system DB (Replication: txn-log "table-reload" marker to extend copyApply to the system DB (#480 follow-up) #489) and Replicated hdb_analytics floods system transaction logs and spins a worker in native code; system base-copy wedges until worker recycle #480 closes.
• Copy CPU no longer scales with per-record decode/re-encode.

🤖 Filed by Claude on behalf of Kris.