is it possible to dynamically create queues and workers?

[jimmy-o]

I am currently using express.js with bullMQ.
In my app's entry point: index.ts, I am calling the function:

initializeWorker();

const serverAdapter = new ExpressAdapter();
serverAdapter.setBasePath('/dashboard');
createBullBoard({
    queues: [new BullMQAdapter(processingQueue)],
    serverAdapter,
});

in my initializeWorker.ts, I am declaring the workers.

const commonRedisConfig = {
    host: process.env.REDIS_HOST,
    port: 6379,
    tls: {},  
    maxRetriesPerRequest: null, 
};

const queueRedisConfig = {
    ...commonRedisConfig,
    retryStrategy: function (times: number) {
        return Math.max(Math.min(Math.exp(times), 20000), 1000);
    },
    enableOfflineQueue: false,
};
const queueConnection = new IORedis(queueRedisConfig);

const workerRedisConfig = {
    ...commonRedisConfig,
    enableOfflineQueue: true
};

const workerConnection = new IORedis(workerRedisConfig);

const queueName = "sharedPublicQueue";

// Use the prefix option with brackets to ensure compatibility with Redis cluster
const queueOptions = {
    connection: queueConnection,
    prefix: '{myapp}'
};

const workerOptions = {
    lockDuration: 180000,
    prefix: '{myapp}',
    concurrency: 10,
    connection: workerConnection
};

const queue = new Queue(queueName, queueOptions);
export const processingQueue = queue;

export const initializeWorker = () => {
    new Worker(queueName, async (job: Job) => {
     // logic to process the jobs
    },
       workerOptions
    );
};

This setup is working fine for me for now on a PROD env. I am using pm2 to start my application:

CMD ["pm2-runtime", "start", "ecosystem.config.js"]

and my config for pm2:

module.exports = {
    apps : [{
        name: "myapp",
        script: "dist/src/index.js",
        //instances: "max", // Adjust based on your needs; "max" for as many instances as CPU cores
        //exec_mode: "cluster", // Enable cluster mode for load balancing across instances
        autorestart: true,
        watch: false,
        max_memory_restart: "1G", // Adjust based on your needs
        env: {
            NODE_ENV: "production",
        },
    }],
};

Currently, my setup, which includes Elasticache, relies on a single shared public queue to manage tasks from multiple clients of my API product. I'm contemplating the feasibility of setting up dedicated queues and workers for each client to manage their API requests more efficiently.

For instance, when User A initiates a task, it's initially processed through the shared public queue. However, I'm interested in whether it's possible to then dynamically allocate a dedicated queue and specific workers for that user based on their unique configurations. This way, should User A make subsequent requests, they would be directed to their personalized queue.

how many max queues can I have? I am also thinking manually create X (the number of client) queue when I start my application. But not sure if this is feasible too.

I'm looking for clarity on whether this approach can be realized effectively. Thank you for any insights or advice you can provide.

[manast]

There is no max limit on the amount of queues that you can have, and you could create that many queues, however the problem is that it gets quite cumbersome to handle if you have one per user for instance and the number of queues gets into the hundreds or thousands. Management and debugging will be quite difficult as you need to navigate through all the queues. In BullMQ Pro we have support for groups which address exactly this issue: https://docs.bullmq.io/bullmq-pro/groups