Too many open Redis Connections while using BullMQ

[VinuthnaGangula]

I am using Redis BullMQ (V5) to implement a Message Queue system to create a Queue for each visitor of our application and process their messages through this Queue so that one visitor doesn't have to wait for other visitors' messages to be processed; and we are using Redis Labs.

I am passing Redis Host, Port details to instantiate a queue. It is observed that the Redis connections created by a queue are not closed after the queue is closed or even obliterated, which is causing spikes in the Redis. Can you suggest how we could close the connections after the queue is done with its jobs?

Here's a snippet of my implementation:

class MessageQueue {
  constructor() {
    this.queueOptions = {
      // Queue Options,
      connection: {
        host: '<redis host>',
        port: '<redis port>',
        tls: true,
        password: '<password>'
      },
    };

    // ...
    let messageQueue = new Queue(queueName, this.queueOptions);
    // Add job to the queue and create worker if there's none.
    // ...
  }
}

[manast]

@VinuthnaGangula The connections are closed when you call the queue.close() or worker.close() method, I wonder if you have a more concrete example when this is not the case.
For the scenario that you are describing you would probably want to reuse Redis connections to keep the number lower by passing a Redis instance to the "connection" option instead of the Redis settings. Also if the number of users is very high you may want to consider the Pro version: https://docs.bullmq.io/bullmq-pro/groups

[VinuthnaGangula]

Hi @manast , thanks for quick reply.

I could see that the Redis connections are persistent because of Worker (Also I have used queue.close() but not worker.close()). Now my problem is that I need the Worker instance to close it and I couldn't find how I could do that(Other than the place where Worker instance is returned on its creation).

Is there a method that could provide the workers of a queue along with their instances so that I could close them?

For the scenario that you are describing you would probably want to reuse Redis connections to keep the number lower by passing a Redis instance to the "connection" option instead of the Redis settings. Also if the number of users is very high you may want to consider the Pro version: https://docs.bullmq.io/bullmq-pro/groups

Thanks for the above recommendations, We'd consider them as well.

[manast]

I am not sure what you are trying to do exactly so I do not understand what issue you have with closing the workers, however I think this blogpost could be helpful for you to better understand how to design your solution: https://blog.taskforce.sh/do-not-wait-for-your-jobs-to-complete/

[VinuthnaGangula]

Hi @manast , Sorry for late reply.

May be I'll explain with an example. Say, We have 100 customers. When a customer triggers a chat, we create a queue for them with a worker attached. Every message from that customer chat is put into the queue created for them and the worker starts processing them one by one.

Since I am not reusing connections and passing Redis host & port instead, the queues & workers are trying to establish connection to the Redis server. As far as we observed, for my implementation, each queue is establishing one connection on instantiation and a worker is establishing two connections. I instantiate a queue to add a job and close it once done. But I never closed the worker since it needs to process the jobs. As the number of customers increased, the queues & workers also increased, which accumulated the Redis connections and caused a spike overtime. For the current scenario, at one point, we have 300 connections created for BullMQ alone, which is more than 250% of what the rest of the application consumes.

Here is the graph that shows the spike:

We are using a Redis Labs with max client connections of 5000.

Currently, we got the situation under control by closing the workers periodically. Other than that, I think I should have reused the Redis Connections. If you see or think there is something else I should have done, please do recommend.

Thank you!

[manast]

Yes, you can try to reuse and manually close unused workers, but I think that as the number of customers increase this will be quite difficult to maintain, thats why I recommended you the groups functionality which matches perfectly with your requirements.

[VinuthnaGangula]

Sure, thank you @manast , it all makes sense now.