Skip to content

fix(pulse): detect dead websockets and back off reconnects - #11097

Open
MichaelUray wants to merge 1 commit into
hcengineering:developfrom
MichaelUray:fix/pulse-ping-timeout
Open

MichaelUray wants to merge 1 commit into
hcengineering:developfrom
MichaelUray:fix/pulse-ping-timeout

Conversation

@MichaelUray

Copy link
Copy Markdown
Contributor

Fixes #10946

Problem and change

HulypulseClient and PresenceClient recreated their 5 minute ping timeout on
every 30 second ping tick, so the timeout was always cleared before it could
fire. HulypulseClient additionally only closed the socket when it was not
OPEN, so a half-open connection that stopped answering pings was never
detected, and both clients reconnected on a fixed 1 second timer.

The dead-connection timeout is now started on connect and reset only when a
message (including pong) is received from the server. When it fires, the
stale socket is detached and closed and a reconnect is scheduled; late events
from the detached socket are ignored, so no second reconnect is started.
Reconnects use exponential backoff capped at 30 seconds with jitter
(getReconnectDelay, shared via @hcengineering/hulypulse-client), reset after a
successful connect. Sending a request no longer restarts the ping timer.

Tests

client.test.ts with fake timers and a fake WebSocket (pings every 30 s, dead socket closed and reopened after 5 min without replies, pongs keep the connection, capped backoff, no reconnect after close). The hulypulse server answers text ping with pong; the presence client relies on the same contract it already expected (the presence server is not part of this repository).

HulypulseClient and PresenceClient recreated their 5 minute ping timeout on
every 30 second ping tick, so the timeout was always cleared before it could
fire. HulypulseClient additionally only closed the socket when it was not
OPEN, so a half-open connection that stopped answering pings was never
detected, and both clients reconnected on a fixed 1 second timer.

The dead-connection timeout is now started on connect and reset only when a
message (including pong) is received from the server. When it fires, the
stale socket is detached and closed and a reconnect is scheduled; late events
from the detached socket are ignored, so no second reconnect is started.
Reconnects use exponential backoff capped at 30 seconds with jitter
(getReconnectDelay, shared via @hcengineering/hulypulse-client), reset after a
successful connect. Sending a request no longer restarts the ping timer.

Fixes hcengineering#10946

Signed-off-by: Michael Uray <michaeluray@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fix ineffective websocket heartbeat timeouts and reconnect stampedes

1 participant