Skip to content

fix(redis): preserve ioredis reconnect ownership - #2350

Closed
og2701 wants to merge 2 commits into
devfrom
fix/dragonfly-resize-reconnect
Closed

fix(redis): preserve ioredis reconnect ownership#2350
og2701 wants to merge 2 commits into
devfrom
fix/dragonfly-resize-reconnect

Conversation

@og2701

@og2701 og2701 commented Jul 22, 2026

Copy link
Copy Markdown
Member

Summary

  • keep ioredis in control of reconnecting after a provider-side connection close
  • use disconnect(true) only when repeated probe failures occur while the client still reports ready
  • add a socket-level regression test covering endpoint loss beyond the old stale threshold and subsequent recovery

Root cause

During an in-place Dragonfly resize, the provider can close an established connection. ioredis starts its normal reconnect loop, but the availability monitor previously treated a reconnect lasting more than five seconds as stale and called disconnect(false) followed by connect(). The explicit non-reconnecting disconnect could cancel the client-owned retry lifecycle and leave the process unable to recover when the endpoint returned.

Impact

Redis-backed features can recover automatically after Dragonfly closes connections during a resize, without restarting the application process.

Verification

  • Redis availability integration suite: 21 passed, 0 failed
  • focused reconnect regression using a real ioredis client and TCP endpoint: passed
  • server typecheck: passed
  • focused Biome check: passed
  • git diff --check: passed

A real staging Dragonfly in-place resize remains the final provider-level validation.


Summary by cubic

Prevent Redis downtime during Dragonfly in-place resize by letting ioredis own reconnects after provider-side closes. Also enables staging deploy for this branch to validate recovery.

  • Bug Fixes

    • Remove stale reconnect override and timestamp logic; do not interrupt ioredis reconnect loops.
    • On repeated failures while status is ready, call disconnect(true) and report connection down until recovery.
    • Add TCP-level regression test that simulates a provider close; verifies recovery and no disconnect(false) usage.
  • CI

    • Add fix/dragonfly-resize-reconnect to STAGING_DEPLOY_BRANCH_ALLOWLIST to allow staging validation.

Written for commit 44c7931. Summary will update on new commits.

Review in cubic

Greptile Summary

This PR keeps Redis reconnect handling under ioredis control. The main changes are:

  • Bug fixes: Removes the stale reconnect timeout and manual reconnect sequence.
  • Improvements: Uses disconnect(true) after repeated probe failures while the client remains ready.
  • Improvements: Adds a socket-level test for endpoint loss and recovery.

Confidence Score: 5/5

This looks safe to merge.

  • No blocking issues found in the changed code.
  • Non-ready clients remain classified as unavailable without interrupting their reconnect loop.
  • Repeated failures while ready still trigger a reconnect through ioredis.

Important Files Changed

Filename Overview
server/src/external/redis/initUtils/createRedisAvailability.ts Removes manual stale reconnect handling and lets ioredis retain ownership of retry scheduling.
server/tests/integration/others/redis/create-redis-availability.test.ts Adds a TCP endpoint recovery test and updates expectations for reconnecting ready clients.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Run Redis probe] --> B{Client ready?}
    B -- No --> C[Report connection down]
    B -- Yes --> D[Send PING]
    D -- PONG --> E[Report available]
    D -- Failure --> F{Failure threshold reached?}
    F -- No --> G[Report unresponsive]
    F -- Yes --> H[Call disconnect true]
    H --> I[ioredis schedules reconnect]
    I --> J{Endpoint restored?}
    J -- No --> C
    J -- Yes --> A
Loading

Reviews (1): Last reviewed commit: "ci: allow dragonfly reconnect staging de..." | Re-trigger Greptile

Context used:

  • Context used - When generating the key changes section of the sum... (source)

@og2701
og2701 marked this pull request as ready for review July 22, 2026 12:07
@og2701
og2701 requested review from ay-rod and johnyeocx as code owners July 22, 2026 12:07
@vercel

vercel Bot commented Jul 22, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

2 Skipped Deployments
Project Deployment Actions Updated (UTC)
checkout Ignored Ignored Jul 22, 2026 12:07pm
landing-page Ignored Ignored Jul 22, 2026 12:07pm

Request Review

@capy-ai

capy-ai Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Capy auto-review is paused for this organization because the usage-cycle auto-review limit has been reached. Increase the limit or turn it off in billing settings to resume automatic reviews.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 3 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Re-trigger cubic

@og2701 og2701 closed this Jul 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant