Replication W3: Reconnect/storm control & resource bounds #433

Workstream W3 of #430 · Foundation · removes the OOM/CPU-spiral class

Status update (2026-07-01)

Partially stale on one detail: subscription-setup retries now have fixed delays in places (scheduled setTimeout retries in subscriptionManager.ts, and a copy-cursor flush-failure backoff landed on the connection side), so "spins with no delay" no longer describes every path. But #327 remains open and the substance stands: there is still no uniform backoff discipline (exponential + jitter + cap) across every layer that can initiate work, and #349 (one-shot intent leaking into a persistent listener) and #288 (per-reconnect TLS context churn) are unchanged. Re-verify #327's exact repro against current main when picking this up.

Summary

Backoff is applied inconsistently across layers, one-shot intents leak into persistent listeners, expensive per-reconnect work amplifies churn into OOM, and oversized payloads are rejected rather than chunked. Together these are the resource-exhaustion bug class under churn.

Root cause / current state

• The connection layer has proper exponential backoff; the subscription-setup layer (subscriptionManager.ts, on the main thread) has only ad-hoc fixed delays — a boot-time DNS failure can still retry-storm (replication: no-backoff subscription-setup retry storm on transient boot-time DNS failure ends in OOM #327, ~1,600 log-lines/s → OOM as originally observed; re-verify current behavior).
• A one-shot forceResubscribe intent binds to a persistent listener, so every table event re-drives a full resubscribe (Replication: 5.0.31 wedge-reconcile spins at 100% CPU re-subscribing an unreachable peer (forceResubscribe leaks into persistent listeners) #349, ~4,850 lines/s → 100% CPU).
• tls.createSecureContext runs per reconnect in monitorNodeCAs, so reconnect churn allocates unboundedly (Investigate: heap corruption / OOM from rapid TLS context churn in monitorNodeCAs during reconnect cycles #288, heap churn/OOM).
• Oversized messages are dropped above MAX_PAYLOAD (checkExcessMessageSize) rather than chunked, so a single large record can't traverse the cluster (the live-WS analogue of the abandoned-NATS NATS MAX_PAYLOAD_EXCEEDED error when cluster cloning large data #266).

Design direction

1. One backoff discipline at every layer that can initiate a connection — subscription-setup gets exponential backoff + jitter + cap, mirroring the connection layer; treat ENOTFOUND/connection errors as retryable-with-delay; never spin the main thread.
2. One-shot intents must not bind to persistent listeners — thread a fromEvent flag so persistent re-fires behave as no-ops.
3. Memoize createSecureContext keyed on CA material so reconnect churn reuses TLS contexts.
4. Memory-bounded admission control — back-pressure on bytes/heap outstanding, not just object counts.
5. Chunk oversized records/payloads through the streaming (blob-style) transport so replication never depends on a single sub-MAX_PAYLOAD message.

Scope

• Backoff + jitter + cap on subscription-setup; no main-thread spin
• fromEvent guard so one-shot resubscribe intents don't bind persistent listeners
• Memoized createSecureContext keyed on CA material
• Bytes/heap-bounded admission control
• Chunked transport for oversized records/messages

Retires / advances

• replication: no-backoff subscription-setup retry storm on transient boot-time DNS failure ends in OOM #327 — no-backoff subscription-setup retry storm on boot DNS failure → OOM
• Replication: 5.0.31 wedge-reconcile spins at 100% CPU re-subscribing an unreachable peer (forceResubscribe leaks into persistent listeners) #349 — forceResubscribe storm (storm/CPU half)
• Investigate: heap corruption / OOM from rapid TLS context churn in monitorNodeCAs during reconnect cycles #288 — heap churn/OOM from TLS context churn during reconnect cycles
• NATS MAX_PAYLOAD_EXCEEDED error when cluster cloning large data #266 — large-clone oversized payload (reframed onto the WS MAX_PAYLOAD path; the NATS-specific issue is stale)

Dependencies

Complements W1. Independent enough to start immediately (several are localized, high-ROI fixes — good Phase-1 quick wins).

Effort / risk

M / low–medium.

Acceptance criteria

• A boot-time DNS failure backs off cleanly (no main-thread spin, bounded log volume).
• Reconnect churn reuses TLS contexts (flat heap).
• A record larger than MAX_PAYLOAD replicates via chunking.

🤖 Filed by Claude on behalf of Kris.