One queue per job type vs multiple job types in the same queue?

[xinluh]

Apologies in advance if this is answered elsewhere and I missed it.

We have some ~100 different types of jobs, unrelated to each other. Previously in bull we were using the named processor pattern and process the jobs in a single queue.

I understand that the named processor was just syntactic sugar, but it is a little unclear to me what are the tradeoffs between 1) continue using the named processor pattern, maybe even with the group feature for more granular control of per job type concurrency, and 2) processing each job type in their own separate queue.

What resources would spawning 100's of queues use and is there limitation on number of queues? What pattern would you recommend using?

[manast]

I do not think spawning hundreds of queues is a good pattern because it will be difficult to manage. You can use 1 queue and just simulate named processors with a switch case if you do not have any other requirements for instance that the jobs should be processed evenly across the different job names. In that case you would need something like the groups' feature in the Pro version:
https://docs.bullmq.io/bullmq-pro/groups

[xinluh]

Thank for confirming my suspicions that spinning up a queue per job type isn't a great idea.

A quibble: in the docs, it almost makes it seem like one job type per queue is recommended, especially with the removal of the syntactic sugar w.r.t named processor. The switch statement seems hacky for IMO a common usecase of having a queue with different job processor logic.

In addition, the typing for Queue, while optional, suggests that it is best used to hold one job type as well (different job types would obviously need different input/output types)

[marbemac]

In addition, the typing for Queue, while optional, suggests that it is best used to hold one job type as well (different job types would obviously need different input/output types)

Agree with this! Currently trying to find a balance re how best to type the queues, workers, and producers, and it's feeling a bit awkward so far...