Architecture for multi chat system with rate limiting for each chat group

[BoKKeR]

Hello! I created a RSS bot which posts links from RSS to telegram channels. The project is quite simple I use a bullMQ for each message to be sent out.

Now the problem is that there are two kinds of API limits I am facing with telegram. One is for too many messages globally, and the other one is too many messages per chat group.

Currently when I hit a 429 too many requests I just pause the queue for all users, and return it after the given time (api gives retry_in 40 seconds).

This is not the greatest solution because:

CHAT A messages: get 429 -> stops processing queue for everyone for 1 minute
CHAT B messages: paused 🛑
CHAT C messages: paused 🛑
CHAT D messages: paused 🛑

I would like to have an option to only pause the jobs that related to the specific group that got rate limited. Instead of stopping the whole queue just because one chat group misbehaved!

CHAT A messages: get 429 -> stops processing queue CHAT A
CHAT B messages: not paused 🟩
CHAT C messages: not paused 🟩
CHAT D messages: not paused 🟩

Currently my jobs look like this:

await this.messagesQueue.add(
        "message",
        {
          chatId: chatId,
          url: url
        },
        {
          removeOnComplete: true
        }
      );

Ideas:

• create a queue for each chat_id, but this would create too many redis connections
• update all pending jobs with same chat_id and add a retry_at_timestamp, only retry these messages if the timestamp has passed, dont pause the global queue
• something else?

Any help is appreciated, I love the project and I am new to redis/bull :)

Currently this is how the queue behaves:

[manast]

This is in fact a common, and quite hard problem to solve, that's why we implemented "groups" in the Pro version https://docs.bullmq.io/bullmq-pro/groups
It seems that the Pro package will not suitable for you though as your project is open-source. In that case, I think the only way to solve it is to have a queue for every chat group and if the number of groups is not too large it may be manageable, Redis supports a very high number of connections (10k by default but it can be increased: https://redis.io/docs/reference/clients/).

[BoKKeR]

Thanks for the answer! I will try to implement dynamic queues