ClusterAllFailedError after upgrading from 4.17 to 5

[nullndr]

Today I updated the bullmq version in one project from 4.17 to 5.7 gradually (From 4.17 to 5.1, from 5.1 to 5.3 and from 5.3 to 5.7).

Everything worked great with the migration but after a bunch of hours the project was running I received the following error:

ClusterAllFailedError: Failed to refresh slots cache.
    at tryNode (/home/admin/node_modules/ioredis/built/cluster/index.js:310:31)
    at /home/admin/node_modules/ioredis/built/cluster/index.js:327:21
    at Timeout.<anonymous> (/home/admin/node_modules/ioredis/built/cluster/index.js:664:24)
    at Timeout.run (/home/admin/node_modules/ioredis/built/utils/index.js:117:22)
    at listOnTimeout (node:internal/timers:571:11)
    at process.processTimers (node:internal/timers:512:7) {
  lastNodeError: Error: timeout
      at timeout (/home/admin/node_modules/ioredis/built/utils/index.js:120:38)
      at EventEmitter.getInfoFromNode (/home/admin/node_modules/ioredis/built/cluster/index.js:661:67)
      at tryNode (/home/admin/node_modules/ioredis/built/cluster/index.js:316:19)
      at EventEmitter.refreshSlotsCache (/home/admin/node_modules/ioredis/built/cluster/index.js:335:9)
      at /home/admin/node_modules/ioredis/built/cluster/index.js:168:22
}
[2024-05-23T11:00:04.032Z] Node encountered an unhandled promise rejection
Error: None of startup nodes is available
    at EventEmitter.closeListener (/home/admin/node_modules/ioredis/built/cluster/index.js:160:35)
    at Object.onceWrapper (node:events:631:28)
    at EventEmitter.emit (node:events:529:35)
    at /home/admin/node_modules/ioredis/built/cluster/index.js:580:18
    at processTicksAndRejections (node:internal/process/task_queues:77:11)
    at runNextTicks (node:internal/process/task_queues:64:3)
    at listOnTimeout (node:internal/timers:538:9)
    at process.processTimers (node:internal/timers:512:7)
Promise {
  <rejected> Error: None of startup nodes is available
      at EventEmitter.closeListener (/home/admin/node_modules/ioredis/built/cluster/index.js:160:35)
      at Object.onceWrapper (node:events:631:28)
      at EventEmitter.emit (node:events:529:35)
      at /home/admin/node_modules/ioredis/built/cluster/index.js:580:18
      at processTicksAndRejections (node:internal/process/task_queues:77:11)
      at runNextTicks (node:internal/process/task_queues:64:3)
      at listOnTimeout (node:internal/timers:538:9)
      at process.processTimers (node:internal/timers:512:7)
}

From the path this totally looks like an ioredis error, so I upgraded my ioredis 5.3 version to the latest 5.4 but the problem has been raised again.

I am pretty sure this is something regarding bullmq 5 since I have never had this error in the months my project was running with bullmq 4.

I'm running bullmq on an AWS EC2 (node v18) connected to an AWS MemoryDB in cluster mode, I can share my code configuration:

Connection configuration

const baseRedisOptions: RedisOptions = {
  maxRetriesPerRequest: null,
  enableReadyCheck: false,
  showFriendlyErrorStack: true,
  retryStrategy: (t) => t * t * 1000,
  tls:
    process.env.NODE_ENV === "production" && Env.get("CLUSTER_MODE_ENABLED")
      ? {
          /*
           * This is the same as the `--skipverify` flag in redli.
           * In production we must have a strong certificate, with a know authority.
           */
          rejectUnauthorized: false,
        }
      : undefined,
};

const queueRedisOptions: RedisOptions = {
  ...baseRedisOptions,
  enableOfflineQueue: false,
};

const workerRedisOptions: RedisOptions = {
  ...baseRedisOptions,
  enableOfflineQueue: true,
  maxRetriesPerRequest: null,
};

export const queueRedisConnection =
  process.env.NODE_ENV === "production" && Env.get("CLUSTER_MODE_ENABLED")
    ? new Redis.Cluster([{ host: Env.get("REDIS_HOST"), port: 6379 }], {
        clusterRetryStrategy: () => null,
        enableReadyCheck: true,
        enableOfflineQueue: false,
        showFriendlyErrorStack: true,
        redisOptions: {
          tls: {
            checkServerIdentity: () => undefined,
          },
          username: Env.get("REDIS_USERNAME"),
          password: Env.get("REDIS_PASSWORD"),
        },
      })
    : new Redis(Env.get("REDIS_HOST"), queueRedisOptions);

export const workerRedisConnection =
  process.env.NODE_ENV === "production" && Env.get("CLUSTER_MODE_ENABLED")
    ? new Redis.Cluster([{ host: Env.get("REDIS_HOST"), port: 6379 }], {
        clusterRetryStrategy: () => null,
        enableReadyCheck: true,
        enableOfflineQueue: true,
        showFriendlyErrorStack: true,
        redisOptions: {
          tls: {
            checkServerIdentity: () => undefined,
          },
          username: Env.get("REDIS_USERNAME"),
          password: Env.get("REDIS_PASSWORD"),
        },
      })
    : new Redis(Env.get("REDIS_HOST"), workerRedisOptions);

export const flowRedisConnection =
  process.env.NODE_ENV === "production" && Env.get("CLUSTER_MODE_ENABLED")
    ? new Redis.Cluster([{ host: Env.get("REDIS_HOST"), port: 6379 }], {
        clusterRetryStrategy: () => null,
        enableReadyCheck: true,
        enableOfflineQueue: false,
        showFriendlyErrorStack: true,
        redisOptions: {
          tls: {
            checkServerIdentity: () => undefined,
          },
          username: Env.get("REDIS_USERNAME"),
          password: Env.get("REDIS_PASSWORD"),
        },
      })
    : new Redis(Env.get("REDIS_HOST"), queueRedisOptions);

Here is how I handle errors and interrupts

const closeAll = async () => {
  await Promise.allSettled([closeWorkers(), closeQueues(), closeFlows()]);
  /**
   * The database and the redis connection must be closed after all workers complete.
   */
  await Promise.allSettled([disconnectDb(), disconnectRedis()]);
  await notify({
    type: NotifyType.JobStopped,
    pid: process.pid,
  });
};

/**
 * For development, Ctrl-C
 */
process.on("SIGINT", async (signal) => {
  await runOnce(async () => {
    logger(`I received a \x1b[1m\x1b[96m${signal}\x1b[0m signal.`);
    await closeAll();
    process.exit(130);
  });
});

/**
 * For preview and production
 */
process.on("SIGTERM", async (signal) => {
  await runOnce(async () => {
    logger(`I received a \x1b[1m\x1b[96m${signal}\x1b[0m signal.`);
    await closeAll();
    process.exit(143);
  });
});

process.on("unhandledRejection", async (error, promise) => {
  await runOnce(async () => {
    logger(logger.err("Node encountered an unhandled promise rejection"), {
      error,
      promise,
    });
    await closeAll();
    process.exit(1);
  });
});

process.on("uncaughtException", async (error, origin) => {
  await runOnce(async () => {
    logger(logger.err(`Node encountered an uncaught exception (${origin})`), {
      error,
    });
    await closeAll();
    process.exit(1);
  });
});

process.on("uncaughtExceptionMonitor", async (error, origin) => {
  await runOnce(async () => {
    logger(
      logger.err(`Node encountered an uncaught exception monitor (${origin})`),
      {
        error,
      },
    );
    await closeAll();
    process.exit(1);
  });
});

Do you have any idea about this? Until now my workers have been gracefully shut-down, so I simply restarted them.

[manast]

The thing is that BullMQ 5 uses a newer version of IORedis, so it is quite likely the issue has been introduced in IOredis as a regression, the question is in which scenario this happens so that we can report it in IORedis repo.

[nullndr]

I was using bullmq 4.17 with ioredis 5.3.2. I rolled back bullmq from 5.7 to 5.1 and ioredis from 5.4.1 to 5.3.2 and everything looks fine again.

[manast]

Some background: redis/ioredis@aa9c5b1

[manast]

Some hints may come from this error as well: Error: None of startup nodes is available
So it is also possible there is an issue in the cluster itself... which before was for some reason not being reported by IORedis. I think more investigation is required here before we can conclude there really is an issue in IORedis or BullMQ.