fix(replication): recovery nets miss a subscription wedged at connected:true / WAITING with no receive progress #810

Summary

A replication subscription that parks at connected: true + RECEIVING_STATUS_WAITING with no receive progress is invisible to both reconcile recovery nets, so the wedge is permanent. The stated design intent — "guaranteeing the wedge can never be permanent" (subscriptionManager.ts L119-121) — is not met for this shape.

Observed in production: one user database stopped replicating on two receiving nodes simultaneously and stayed dead for ~21 hours until the containers were restarted manually. Throughout, cluster_status reported connected: true on every leg, get_status reported all components healthy, and the system database kept replicating normally on the same connections — so every health signal read green.

Root cause

Both nets exclude this state:

export function isReceiveStalled(status, now, thresholdMs) {
	return (
		status != null &&
		status.status === RECEIVING_STATUS_RECEIVING &&   // <- only Receiving
		status.lastReceivedTime > 0 &&
		now - status.lastReceivedTime >= thresholdMs
	);
}

findStalledReceivingNodeUrls was deliberately scoped to the ping-alive base-copy stall (#453), where a follower parks at Receiving. A steady-state (non-copy) subscription that stops receiving parks at WAITING instead, and nothing covers it.

Confirmed by elimination

Verified against the live wedged nodes; every other gate was satisfied, so only the status check can have failed:

• entry.connected was true for the full ~21h → findWedgedNodeUrls excluded.
• isDesired / shouldReplicateFromNode was true: hdb_nodes self row present with replicates: true, peer rows replicates: true, replication.databases: "*", local database present. (selfNodeReplicates correctly uses getSync, so the documented block-cache/Promise-eviction hazard was not in play — though note the system database was 6.7 GB here, well inside that danger zone.)
• lastReceivedTime > 0 (frozen at the failure instant) and age ≫ RECEIVE_STALL_THRESHOLD_MS (21h vs 15min) → both satisfied.
• The once-per-episode guard (receiveStallReconnectAt == null || lastReceivedTime > receiveStallReconnectAt) was satisfied — the only prior stalled-path kicks on that container were weeks earlier, and lastReceivedTime had advanced far past them since.
• Negative check: the stalled path's distinct log line (Reconciling replication subscriptions stalled connected:true with no receive progress) appeared zero times across ~21h (~84 reconcile opportunities at the 15-min cadence), but did appear on earlier dates on the same container — so the detector is live code, and stalledByUrl was genuinely always empty.

Reproduced independently on two receiving nodes, wedged within ~12 seconds of each other.

Trigger

The sender crashed and restarted mid-stream. In its final seconds it emitted a record whose typed-structure definition never arrived, and both receivers threw during value decode:

[error] [replication]: Error decoding replication message, record id: <redacted>
  typed structures for current decoder[]        <- empty
  structures for current decoder[[...]]

The decode handling itself behaved correctly (logged, skipped, advanced the watermark). The connection then dropped 1006, reconnected ~30s later for both databases — and system resumed while the user database never delivered another record.

Note the trigger is generic: any sender crash under load can produce an undecodable record, so this is not specific to one deployment.

Why the obvious fix is wrong

RECEIVING_STATUS_WAITING is the normal idle/between-batches state — a healthy leg with a quiet source sits there too, with an equally old lastReceivedTime. Simply relaxing the status check would force-reconnect every legitimately idle subscription every 15 minutes.

A correct fix needs to distinguish "idle because the sender has nothing" from "wedged while the sender has data".

Constraint on the design

The receiver cannot currently make that distinction locally. The shared status buffer carries no advertised sender head — the positions are CONFIRMATION_STATUS, RECEIVED_VERSION, RECEIVED_TIME, SENDING_TIME, LATENCY, RECEIVING_STATUS, BACK_PRESSURE_RATIO, BLOB_FAILURE_COUNT, LAST_BLOB_FAILURE_TIME, CONNECTION_STATE, LAST_LIVENESS_TIME, LAST_ERROR_CODE, LAST_ERROR_TIME. LAST_LIVENESS_TIME does not help — pongs keep advancing on a wedged-but-open socket exactly as on an idle one.

The sender does hold the discriminating signal: during the incident the sender's own lastCommitConfirmed for both peers stayed frozen at the failure instant while it held newer commits. That is already tracked per peer per database.

Candidate directions

1. Sender-side unconfirmed-commit stall detection — sender detects "this peer has unconfirmed commits older than threshold" and terminates its side, forcing the receiver to reconnect and resubscribe. Uses data already tracked; the sender is the only party that can tell idle from wedged. Adds a recovery net on the sending side.
2. Advertise the sender head to the receiver (new shared-status position / protocol field) so the receiver can compare its applied watermark against the peer's head and keep recovery where forceReconnect already lives. Requires a protocol addition.
3. Targeted recovery at the decode-failure site — mark the connection for forced reconnect/resubscribe when a value decode throws. Cheap and fixes the observed trigger, but leaves the general blind spot open for any other cause of a WAITING park.

Directions 1 and 2 close the gap; 3 only addresses the observed trigger.

Impact

• Silent: no error after the initial decode failure, and every health surface reads green.
• Unbounded duration: no mechanism recovers it; only a manual restart of the receiving node does.
• Correctness-visible to clients: readers on affected nodes served stale data indefinitely while believing they were current.

Environment

• Reproduced on harper-pro 5.1.26; the gap is present in current main (verified at replication/subscriptionManager.ts L340-351 and L263).
• 3-node mesh, replication.databases: "*", RocksDB storage.