Add per-origin closed timestamp floors for compact, cross-node subscription resume #922

Problem

Subscriptions need a compact checkpoint that can resume on another replica without silently missing delayed transactions. Clients may receive only a small fraction of the transaction log after table/key/row/event filtering, so the timestamp of the last matching message is also a poor progress checkpoint: a quiet subscription can repeatedly scan a long interval it already examined.

The proposed contract is a common timestamp with an overlap window, plus explicit exceptions for origins whose progress cannot be proven within that window. Healthy, caught-up origins need no individual client cursor. Duplicate delivery is acceptable; silent omission is not.

This is a new capability and a design/implementation proposal, not a claim that the current scalar subscription API provides this guarantee.

Ownership within the replication epic

Parent: Replication W4: Per-origin transaction-log convergence (keystone), within Replication reliability & scale — foundations, adaptive topology, and locking.

Keep W4's per-origin progress internally. This issue adds a completeness guarantee and compresses the client checkpoint; it does not replace the internal vector with an unchecked scalar, require a socket per origin, or block the rest of W4 on this feature.

Coordinate these interfaces before implementation:

• W4: portable origin identity/incarnation, per-origin replay positions and transitive propagation.
• W2: Cursor correctness & divergence detection: gap-free applied progress. A cursor that advances while marking data diverged must not certify complete subscription history.
• W11: Code organization, protocol versioning & decode-loop safety: extend the existing capability registry and ordered protocol handling. Capability negotiation already exists; do not treat the older workstream description as current code.
• W13: Base-copy & catch-up path: snapshot/replay/live handoff and explicit resynchronization when history is unavailable.
• W8: Observability & metrics pipeline: expose why progress is held, exception count/age, replay work, and fallback reasons.

Initial scope is RocksDB with preserved end-to-end origin identity. Legacy peers must negotiate down or decline the new guarantee. W12: Hierarchical aggregation / re-origin relay mode needs an explicit translation of the guarantees before it can participate.

Proposed invariant

Use an exclusive floor throughout: a certified floor F covers keys strictly below F; replay includes keys equal to the boundary.

There are three distinct stages:

1. Origin-closed floor: origin O guarantees that no new transaction in this history incarnation can subsequently commit with txnLogKey < F. Every earlier committed transaction below F is covered by an identified log barrier/prefix.
2. Replica-applied floor: a replica has applied that prefix without gaps, with the required durability and blob availability. Seeing a heartbeat, receiving bytes, or observing a maximum transaction timestamp is insufficient.
3. Subscription-processed floor: the subscription has evaluated the complete covered history for its scope and placed all qualifying events before the checkpoint in delivery order. A filter rejection advances processing progress; it does not require a data message.

A floor advertisement needs origin/incarnation, F, scope, and an unambiguous barrier or prefix identity. The exact encoding is open. A receiver-local physical offset or receipt timestamp is not portable across replicas.

Compact checkpoint

Illustrative shape, not a committed public API:

{
  version: 1,
  history: historyIdentity,
  scope: subscriptionScopeIdentity,
  time: 200,
  overlap: 5,
  exceptions: { originC: 120 }
}

Here the common replay boundary is S = time - overlap = 195. An origin can be omitted from exceptions only when its subscription-processed floor is at least S. C resumes inclusively at its own proven floor, 120. With all origins caught up, the exception map is empty.

The timestamp is server-issued checkpoint progress, not the last matching event's timestamp. The overlap is a replay/encoding policy, not an assumed bound on network delay or transaction duration.

An exception means unproven progress, not just disconnection: long transactions, slow apply, missing blobs, clock skew, or consumer backlog can also cause one. Add an exception before advancing S past that origin's proven floor; remove it only after progress catches up. If an origin has no usable floor, preserve an explicit unknown state and use retained-history replay or resynchronization rather than guessing a timestamp.

Persist the complete checkpoint atomically. If wire updates are sparse/delta encoded, define how the client reconstructs and acknowledges a complete checkpoint so a new common time cannot be paired with an old exception set.

Work required

1. Timestamp closure at the producer (rocksdb-js + core)

Design a race-free operation that samples/reserves F and accounts for every outstanding local write that could commit below F, across worker threads and other supported writers of the same database. Cover timestamp reservation before native transaction creation, explicit timestamp adoption, native commit queues, retries whose log batch is already durable, aborts, and worker death.

A later transaction must not acquire an old local-origin key below an advertised floor. Persist the issuance floor before publishing it, or invalidate the history incarnation on recovery. Replicated origin keys and record/LWW versions remain unchanged; neither should raise the local origin's clock implicitly.

The implementation should hold back certification or retain an exception for a long transaction, not delay ordinary message delivery until every origin catches up. Tracking only currently open native handles is insufficient unless core can prove no earlier timestamp can subsequently be installed.

2. Ordered certificates and transitive apply (harper-pro)

Carry a closed-floor certificate behind its covered transactions through the replication data/apply path. A relay must not mint completeness from its own receipt time or forward usable progress ahead of the covered prefix. Handle asynchronous message processing, partial transactions, alternate routes, gaps, and outstanding blobs.

Use existing per-peer connections and negotiate an additive capability. Keep transport liveness pings distinct from completeness. Progress must advance on an idle origin too, without requiring synthetic user writes.

3. Filter-independent subscription progress (core + transports)

Add per-origin processed progress to replay and live delivery, including table/key exclusions, row/event filters, and zero-match intervals. Emit periodic progress-only checkpoints and optionally piggyback them on data messages. They must obey queue ordering/backpressure, transaction completion, cancellation, and client acknowledgement/persistence semantics.

Bind the checkpoint to the database/history, relevant origin membership, subscription scope/filter semantics, and authorization context. Arbitrary changed or stateful filters cannot inherit a previous evaluation guarantee without an explicit application contract. Do not expose unauthorized origin/activity details through progress metadata.

4. Resume admission, retention, and seeks (core + harper-pro + rocksdb-js where necessary)

On another replica, validate the checkpoint identities and retention for every applicable boundary. A target lacking the required prefix must catch up through a bounded path or return an explicit resync requirement; it must not silently accept an incomplete history. Register new origins and handle joins/leaves without assuming the current node-name list proves historical coverage.

Use indexed per-origin seeks and ordinary timestamp filtering only where the new floor invariant makes them safe. Preserve each origin's replay boundary during replay/live handoff; advancing it to the largest observed event timestamp would recreate the out-of-order omission.

Investigate whether durable origin-issued floor records should also index physical boundaries and establish scan stop conditions. Receiver-local delayed-transaction markers alone cannot establish cross-node safety, and a time window does not bound bytes or entries scanned.

Coordinate history invalidation with restore_backup rolls a database back in time with no epoch, and retention admission with Consume the audit staleness floor in Table.subscribe. Define distinct handling for database restore/reclone, origin replacement, and subscription/membership changes rather than overloading an unspecified epoch.

Current source and reusable pieces

Inspected 2026-09-26: harper-pro 5bcdccae8474, its pinned core/harper main b89df2ea7810, and rocksdb-js main c30cdef27864.

• Native transactions have a timestamp before append and support constrained timestamp adoption: transaction_handle.cpp, SetTimestamp. Core can install a timestamp when it lazily creates the native transaction: DatabaseTransaction.ts.
• RocksTransactionLogStore.getRange already provides per-origin starts and exact-boundary machinery. Table.subscribe replay and live broadcast still use scalar timestamp comparisons.
• Replication already sends sequence updates after skipped records and maintains blob-gated durable progress. Reuse those concepts, but neither is an origin-closed timestamp certificate.
• protocolCapabilities.ts is the extension point. recordLockFreshness.ts provides precedent for confirming an origin barrier at apply time; its lock-specific barrier does not close future timestamp issuance.
• Prototype compact subscription resume checkpoints remains a separate internal experiment, with useful adversarial replay cases. Subscription events can identify their origin node with includeOrigin remains separate too; this issue neither retargets it nor makes exposing every origin the client contract.

Acceptance and implementation order

1. Specify the origin closure/barrier identity and prove it with deterministic multi-worker transaction tests, including delayed commit, pre-native reservation, abort/retry, restart, and backward clock steps.
2. Implement certificate transport/apply on top of W4/W2, with capability tests and no per-origin socket multiplication.
3. Add complete subscription checkpoints, progress-only transport events, and cross-node resume admission.
4. Exercise three-node direct/transitive failover with different arrival orders, an unapplied barrier, a gap/blob failure, a long transaction on a ping-alive origin, disconnect/reconnect, new/replaced origins, restore, and retention loss. No qualifying event may be silently omitted; duplicates are allowed.
5. Demonstrate that a healthy idle/filtered cluster produces empty exception maps and advancing checkpoints. Resume after a long zero-match interval must not rescan the entire silent interval when the certified boundary remains retained.
6. Test interruption between data delivery, progress delivery, and checkpoint persistence; partial transactions; replay/live races; and exception insertion/removal. Confirm invalid or unsupported checkpoints fail explicitly.
7. Benchmark write-path overhead, certificate traffic, per-subscription bookkeeping, token size, and replay bytes/entries at 5–20 origins and a larger transitive topology. Include silent, bursty, filtered, skewed, and disconnected workloads. Share database/origin progress rather than rescanning history per subscriber.

Open decisions: ownership of timestamp reservations across JS/native boundaries; durable certificate encoding; cadence/default overlap; stable history/membership/scope identity; and bounded fallback policy. The investigation establishes the implementation seams, not a proof that the complete protocol already works.

Priority: P3, Feature. W4 is P2, but this new client capability has no verified release commitment or specific scheduled roadmap prerequisite. Leave the milestone unset until sequencing is agreed.

— KrAIs (GPT-6)