[Questions] Rabbit HA Cluster: shall classic durable queues be avoided?

[YuryHrytsuk]

Community Support Policy

• I have read RabbitMQ's Community Support Policy
• I run RabbitMQ 4.x, the only series currently covered by community support
• I promise to provide all relevant information (versions, logs from all nodes, rabbitmq-diagnostics output, detailed reproduction steps)

RabbitMQ version used

4.1.2

Erlang version used

27.3.x

Operating system (distribution) used

Ubuntu, Docker Swarm (StackS)

How is RabbitMQ deployed?

Community Docker image

rabbitmq-diagnostics status output

...

Logs from node 1 (with sensitive values edited out)

...

Logs from node 2 (if applicable, with sensitive values edited out)

...

Logs from node 3 (if applicable, with sensitive values edited out)

...

rabbitmq.conf

See https://www.rabbitmq.com/docs/configure#config-location to learn how to find rabbitmq.conf file location

# https://www.rabbitmq.com/docs/cluster-formation#peer-discovery-configuring-mechanism
cluster_formation.peer_discovery_backend = classic_config

cluster_formation.classic_config.nodes.1 = rabbit@rabbit-node01_rabbit01
cluster_formation.classic_config.nodes.2 = rabbit@rabbit-node02_rabbit02
cluster_formation.classic_config.nodes.3 = rabbit@rabbit-node03_rabbit03

# explicitly set cluster name so that all nodes share the same name
# see https://github.com/rabbitmq/rabbitmq-server/discussions/11355
# and https://groups.google.com/g/rabbitmq-users/c/A8v8XSmnCnA/m/4ZpoTtEWAQAJ
# important for grafana dashboard to work properly
cluster_name=osparc.local

## Sets the initial quorum queue replica count for newly declared quorum queues.
## This value can be overridden using the 'x-quorum-initial-group-size' queue argument
## at declaration time.
# https://www.rabbitmq.com/docs/quorum-queues#quorum-requirements
quorum_queue.initial_cluster_size = 3

# Extract proper client ip when behind a proxy (e.g. haproxy)
# https://www.rabbitmq.com/docs/networking#proxy-protocol
# WARNING: this forces clients to use a proxy (direct access to nodes does not work)
proxy_protocol = true

default_queue_type = quorum

Steps to deploy RabbitMQ cluster

1. Clone https://github.com/ITISFoundation/osparc-ops-environments/
2. Go to ./services/rabbit
3. Add env file
4. Run make start-cluster

Steps to reproduce the behavior in question

1. Create a 3-node rabbitmq cluster
2. Start an application that uses and relies on durable classic queue
3. Stop RabbitMQ Node which hosts durable classic queue mentioned above
4. See application going down because queue is not accessible

advanced.config

No response

Application code

No response

Kubernetes deployment file

No response

What problem are you trying to solve?

I want to have a highly available RabbitMQ setup which is robust to 1 node going down (we have 3 in total). By robust I mean that my applications that rely on rabbitmq and communicate over it, keep running even when 1 rabbitmq node goes down.

However, I noticed that my application gets stuck if 1 RabbitMQ node goes down. I presume the culprits are classic durable queues that are stuck to that node.

Shall one avoid durable classic queues if high availability is the goal (aka being robust to 1 node going down in 3-nodes cluster setup)?

[MirahImage]

If your goal is high availability, you should be using quorum queues (or streams). Those are our highly available types.

[YuryHrytsuk]

Thanks for your answer!

Could you also comment on my question from the description?

Shall one avoid durable classic queues if high availability is the goal (aka being robust to 1 node going down in 3-nodes cluster setup)?

[MirahImage]

Classic queues are NOT highly available. This has nothing to do with the durability, and everything to do with the fact that they're not highly available.

[YuryHrytsuk]

If data-safety is not priority, it is fine to have transient classic queue and lose data when rabbitmq node goes down (but at least queue is available if there are other rabbitmq nodes running)

The problem I want to highlight and verify I understand it correctly is:

Using durable classic queues has a fundamental flaw: if rabbitmq node goes down, the durable classic queue is no longer available (even though other rabbitmq nodes of the cluster are running happily) and all workloads relying on it are stuck until rabbitmq node is back. This can be solved by using quorum queues or making classic queue transient (non-durable)

[mkuratczyk]

If data safety is not important but availability is (a common RPC scenario for example) then using exclusive queues is often a good option. Quite likely, you will also like the local-random-exchange. Take a look at the scenario described here: https://www.rabbitmq.com/docs/local-random-exchange