Inter-node Communication Buffer warning

[bdaoudtdc]

Describe the bug

We have a 3 node cluster.
We see this message in the logs
[warning] <0.646.0> rabbit_sysmon_handler busy_dist_port <0.1287.0> [{name,delegate_management_1},{initial_call,{delegate,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.874>,unknown}

According to the docs ..

Inter-node traffic between a pair of nodes uses a TCP connection with a buffer known as the inter-node communication buffer. Its size is 128 MB by default. This is a reasonable default for most workloads.

We believe we have a very light workload , however this message came up few times so we increased the buffer to 512 MB , but we still see the message.

To Reproduce

It happens randomly ..

Version and environment information

RabbitMQ: [3.8.12]
RabbitMQ Cluster Operator: [1.7.0 ]
Kubernetes: [e.g. 1.17.13]

[ansd]

Converting this to a discussion.

[ansd]

Relevant docs in https://www.rabbitmq.com/networking.html#distribution-port-buffer-limit and http://erlang.org/doc/man/erl.html#+zdbbl.

@bdaoudtdc our top suspects are:

1. Large message payloads, or
2. High network latency, or
3. Low throughput (packet loss or shedding perhaps)

Thanks @gerhard @lhoguin.

[gerhard]

To continue this conversation, this will help: https://grafana.com/grafana/dashboards/11352

This is normal in a 3-node cluster running RabbitMQ 3.8 with low to moderate load: http://grafana.gke.rabbitmq.com/d/d-SFCCmZz/erlang-distribution?orgId=1&from=now-24h&to=now

[rauanmayemir]

Ever since upgrading my 3-node cluster to 4.1.0, I started seeing this all too often. I've also set default_queue_type = quorum, so now I occasionally lose delayed messages, my queues can crash because nodes lose their membership, etc, etc.

When I started using streams together with queues, I was losing like 5% of jobs guaranteed. Seems like this might be the reason.

My graph is in the kilobytes range:

[mkuratczyk]

Please start a new discussion and describe the situation in more detail. Off the top of my head, useful information would include:

1. what version did yo upgrade from?
2. you say you changed the default queue type, but does it mean you actually have more quorum queues now? did you migrate any queues? at what point?
3. What do you mean by "I was losing like 5% of jobs guaranteed"? What do it mean to "lose a job"?
4. What do you mean by "nodes lose their membership"?
5. What's in the logs? (attach log files)
6. Some overview of the workload: throughput, number of queues, etc

[mkuratczyk]

and how long had you been running the cluster before the upgrade? A potential confusion here could be that you assume something happened because of the upgrade, while in fact it's more about a cluster restart: queue leaders change, the placement of publishers/consumers vs queue members/leaders changes, etc.

[rauanmayemir]

I upgraded from something like 3.14 to 4.10, and it’s been couple weeks already.

Let me gather some logs and diagnostics and get back in a new thread.

[rauanmayemir]

Oh, before the upgrade it was about couple years. The biggest impact to reliability I see in default_queue_type change and I also believe I might have messed up ‘lazy’ settings.

[mkuratczyk]

lazy setting is ignored since 3.12 https://www.rabbitmq.com/blog/2023/05/17/rabbitmq-3.12-performance-improvements