You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the bug
RabbitMQQueue dials the AMQP connection once on init in queue_rabbitmq.go. gocloud's rabbitpubsub re-opens channels but not connections, so once the connection dies, every publish from that process fails with pubsub (code=FailedPrecondition): Exception (504) Reason: "channel/connection is not open" until someone restarts the pod.
Connections happen to die in normal operation: managed RabbitMQ (CloudAMQP, Amazon MQ) restarts nodes for maintenance and upgrades routinely, plus missed heartbeats and network blips.
To Reproduce
Run any Outpost service against RabbitMQ
Drop the connection from the broker side: rabbitmqctl close_all_connections "test" (or just restart the broker)
Publish anything
Every publish now fails with the 504 above until the process fully restarts.
Expected behavior
The connection is re-established after a drop, like other AMQP clients handle it, and publishing resumes without a process restart.
Additional context
During our testing:
staging: the API service's deliverymq publisher wedged over a weekend. Every publish failed with failed to enqueue delivery task, so nothing was delivered at all until a manual restart.
production: one delivery pod's logmq publisher wedged for six days. Deliveries kept working (HTTP was fine) but every attempt log from that pod was lost failed to publish attempt log. The delivery message is ACKed even when the log publish fails, so those attempt records are gone permanently: missing delivery history, and every failed delivery left an orphaned retry task spinning forever on the no prior attempt found in logstore path (the retrymq limit & DLQ #663 loop).
Describe the bug
RabbitMQQueue dials the AMQP connection once on init in queue_rabbitmq.go. gocloud's rabbitpubsub re-opens channels but not connections, so once the connection dies, every publish from that process fails with
pubsub (code=FailedPrecondition): Exception (504) Reason: "channel/connection is not open"until someone restarts the pod.Connections happen to die in normal operation: managed RabbitMQ (CloudAMQP, Amazon MQ) restarts nodes for maintenance and upgrades routinely, plus missed heartbeats and network blips.
To Reproduce
rabbitmqctl close_all_connections "test"(or just restart the broker)Every publish now fails with the 504 above until the process fully restarts.
Expected behavior
The connection is re-established after a drop, like other AMQP clients handle it, and publishing resumes without a process restart.
Additional context
During our testing:
failed to enqueue delivery task, so nothing was delivered at all until a manual restart.failed to publish attempt log. The delivery message is ACKed even when the log publish fails, so those attempt records are gone permanently: missing delivery history, and every failed delivery left an orphaned retry task spinning forever on theno prior attempt found in logstorepath (the retrymq limit & DLQ #663 loop).