Skip to content

fix(mqs): redial dropped RabbitMQ connections instead of wedging until restart#999

Open
LendritIbrahimi wants to merge 1 commit into
hookdeck:mainfrom
LendritIbrahimi:lendritibrahimi/fix/rabbitmq-reconnect
Open

fix(mqs): redial dropped RabbitMQ connections instead of wedging until restart#999
LendritIbrahimi wants to merge 1 commit into
hookdeck:mainfrom
LendritIbrahimi:lendritibrahimi/fix/rabbitmq-reconnect

Conversation

@LendritIbrahimi

Copy link
Copy Markdown

This fixes the issue where a dropped RabbitMQ connection permanently wedged the queue: the connection was dialed once via sync.Once, so after a broker restart or network blip every subsequent publish failed against the closed connection until the whole pod was restarted. (explained in this issue #998)

The implementation replaces the sync.Once with a mutex-guarded ensureConnected that lazily redials whenever the current connection is nil or closed, reopening the topic on the new connection (the stale topic is shut down in the background). Publish additionally retries once: if a publish fails and the connection turns out to be lost, it redials and re-attempts before surfacing the error.

If the redial itself fails, the original publish error is returned rather than the dial error, so callers see the real failure. Subscribe goes through the same path, so consumers created after a drop also get a fresh connection.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant