Skip to content

Widening test loses its partial-coverage precondition during B's SIGTERM drain (subscriptionCheckpointResume:366) #1258

Closed

Description

@kriszyp

Problem

The cluster test "a route widened from partial coverage runs a full base copy…" in integrationTests/cluster/subscriptionCheckpointResume.test.mjs times out at :366 on main. It started with Sync Core (#1211), which brought in Drain in-flight work and close cleanly on SIGTERM/SIGINT/SIGQUIT (harper#3943). It failed 4 of 6 Cluster 3/6 legs in main run 41663637598 and 11 of 15 local runs. PR CI runs this shard (Node 26), so most PRs see it too.

Cause

  1. The test simulates a partial route: the fixture's SetRouteCoverage writes fullCoverage: false into B's live [seq, A] row, then the test restarts B.
  2. B's live connection to A keeps applying A's level-2 floor certificates. B builds each entry's coverage bit from its in-memory routeFullCoverage (replicationConnection.ts:7349). Core's updateRecordedSequenceId writes that bit into the row (Table.ts:2786-2789).
  3. Since harper#3943, B keeps running for about 5 s after SIGTERM. When A sends a certificate in that window, B's row goes back to fullCoverage: true before B exits. Before, B exited within milliseconds.
  4. On restart, B sees no widening (replicationConnection.ts:12618-12627) and runs no base copy. It keeps the certifiedFrom from the suite's first test, so the clause certifiedFrom === null never holds.

Evidence: in every failing run (4 CI legs, 11 local runs), A sent exactly one certificate to B between the rewrite and B's exit, and B's restart resumed just below that certificate's floor. No passing run had one. In local runs, a read-only poll of B's row saw fullCoverage flip from false to true 17 ms after A's send.

Fix options

This needs an owner's call. The diagnostics task that found it was authorized for instrumentation only.

  • Test: make the rewrite the last write to the row before B restarts. For example, stop B's replication from A before SetRouteCoverage, or stop B with an immediate exit for this one restart. An immediate exit (threads.shutdownTimeout: 0, or a second SIGTERM) narrows the window to about 100 ms but does not close it.
  • Product: stop applying peer floor certificates once shutdown starts. This is not needed for correctness: a live route writing its own coverage bit is by design. It only matters if another path relies on the durable row staying unchanged during a drain.

Secondary hazard: the held transaction held-copy-bound pins A's floor from :350 to :395, against core's default 30 s storage.maxTransactionOpenTime. Each drained restart of B adds up to 5 s. Passing local runs held it 8.8–20.8 s.

Diagnostics that make the next failure report this directly: Make the subscriptionCheckpointResume widening test report why it timed out (#1259).

Review-Deferral: 3e719e20e8f1bd5d severity=nit category=reliability

🤖 Generated by Anthropic Claude (Opus 5.5) dev-agent; posted via @kriszyp.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Fields

Priority

P1

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions