[Bug]: Regression in test-storybook after upgrading to 8.5.x: Axe is already running error

[akibarika]

Describe the bug

Summary

After upgrading to Storybook 8.5.1, I encountered an issue when running yarn test-storybook in combination with axe-playwright. Specifically, the test runner throws the following error:

page.evaluate: Error: Axe is already running. Use `await axe.run()` to wait for the previous run to finish before starting a new run.

This issue occurs when running tests with --maxWorkers=2 or higher. However, it works fine if I downgrade back to 8.4.7 or if I reduce the number of workers to --maxWorkers=1 in 8.5.1.

Environment Details:

• Storybook Version: 8.5.x

• @storybook/test-runner Version: Latest (installed alongside Storybook 8.5.1)

• Test Runner Command:

yarn test-storybook --url http://127.0.0.1:6006/ --config-dir ./apps/storybook/.storybook --maxWorkers=2

• Temporary Solution: Changing --maxWorkers to 1 avoids the issue:

yarn test-storybook --url http://127.0.0.1:6006/ --config-dir ./apps/storybook/.storybook --maxWorkers=1

Questions:

• Was there a change in how @storybook/test-runner or Storybook 8.5.x handles parallel worker processes?

• Is there a way to ensure compatibility with axe-playwright when running test-storybook with multiple workers?

• Are there any known workarounds or fixes to address this regression?

Reproduction link

no

Reproduction steps

I can reproduce this with this code

yarn concurrently -k -s first -n "SB,TEST" -c "magenta,blue" "yarn http-server dist/storybook/storybook --port 6006 --silent" "yarn wait-on http://127.0.0.1:6006 && yarn test-storybook --config-dir ./apps/storybook/.storybook --maxWorkers=2

//.storybook/test-runner.ts

import type { TestRunnerConfig } from '@storybook/test-runner';
import { getStoryContext } from '@storybook/test-runner';

import { injectAxe, checkA11y, configureAxe } from 'axe-playwright';

/*
 * See https://storybook.js.org/docs/writing-tests/test-runner#test-hook-api
 * to learn more about the test-runner hooks API.
 */
const config: TestRunnerConfig = {
  async preVisit(page) {
    await injectAxe(page);
  },
  async postVisit(page, context) {
    // Get the entire context of a story, including parameters, args, argTypes, etc.
    const storyContext = await getStoryContext(page, context);

    // Apply story-level a11y rules
    await configureAxe(page, {
      rules: storyContext.parameters?.a11y?.config?.rules,
    });

    const element = storyContext.parameters?.a11y?.element ?? 'body';
    await checkA11y(page, element, {
      detailedReport: true,
      detailedReportOptions: {
        html: true,
      },
    });
  },
};

export default config;

System

Storybook Environment Info:

  System:
    OS: macOS 15.2
    CPU: (8) arm64 Apple M1 Pro
    Shell: 5.9 - /bin/zsh
  Binaries:
    Node: 20.15.1 - /private/var/folders/4v/rybl0kls7918cw1lzk1120gw0000gp/T/xfs-f358ab08/node
    Yarn: 3.6.4 - /private/var/folders/4v/rybl0kls7918cw1lzk1120gw0000gp/T/xfs-f358ab08/yarn <----- active
    npm: 10.7.0 - /usr/local/bin/npm
  Browsers:
    Chrome: 131.0.6778.265
    Safari: 18.2
  npmPackages:
    @storybook/addon-a11y: ^8.5.1 => 8.5.1 
    @storybook/addon-designs: ^8.0.4 => 8.0.4 
    @storybook/addon-docs: ^8.5.1 => 8.5.1 
    @storybook/addon-essentials: ^8.5.1 => 8.5.1 
    @storybook/addon-interactions: ^8.5.1 => 8.5.1 
    @storybook/addon-links: ^8.5.1 => 8.5.1 
    @storybook/addon-themes: ^8.5.1 => 8.5.1 
    @storybook/blocks: ^8.5.1 => 8.5.1 
    @storybook/core-server: ^8.5.1 => 8.5.1 
    @storybook/react: ^8.5.1 => 8.5.1 
    @storybook/react-vite: ^8.5.1 => 8.5.1 
    @storybook/test: ^8.5.1 => 8.5.1 
    @storybook/test-runner: ^0.21.0 => 0.21.0 
    chromatic: ^11.24.0 => 11.24.0 
    eslint-plugin-storybook: ^0.11.2 => 0.11.2 
    storybook: ^8.5.1 => 8.5.1 
    storybook-addon-pseudo-states: ^4.0.2 => 4.0.2 

Additional context

• This issue does not occur when using the Storybook dev server (yarn storybook) instead of http-server. The same test works fine when pointed to http://localhost:4400 (the Storybook dev server) with --maxWorkers=2.

• I suspect this might be related to how lifecycle events are handled in @storybook/test-runner for static vs. dynamic Storybook builds in the 8.5.x release.

• Similar issue [Investigation]: Support React 19 #29805 (comment)

[greptile-apps]

** Disclaimer** This information might be inaccurate, due to it being generated automatically
This appears to be a race condition in axe-playwright when running tests in parallel. The issue occurs because axe is being injected multiple times across different workers without proper cleanup. To fix this: 1. Add cleanup in the test-runner config: ts //.storybook/test-runner.ts const config: TestRunnerConfig = { async preVisit(page) { // Clean up any existing axe instance await page.evaluate(() => { // @ts-ignore if (window.axe) delete window.axe; }); await injectAxe(page); } // ... rest of config }; 2. Or alternatively, wrap checkA11y calls in try/catch: ts try { await checkA11y(page, element, { detailedReport: true, detailedReportOptions: { html: true, }, }); } catch (e) { if (!e.message.includes('Axe is already running')) { throw e; } // Wait briefly and retry await new Promise(r => setTimeout(r, 100)); await checkA11y(page, element, { detailedReport: true, detailedReportOptions: { html: true }, }); }

About Greptile

This response provides a starting point for your research, not a precise solution.

Help us improve! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

Edit Issue Bot Settings · Greptile

[andykenward]

I've resolved the issue "Axe is already running error" by enabling a11y manual in the Storybook .storybook/preview.ts|tsx file.

const preview: Preview = {
 globals: {
    a11y: {
      manual: true
    }
  }
};

[akibarika]

@andykenward Awesome, you're a lifesaver! I came across this in the docs but never got around to trying it.

[greptile-apps]

** Disclaimer** This information might be inaccurate, due to it being generated automatically
This appears to be a race condition in how axe-playwright is initialized across multiple workers. The issue is likely in the test-runner's handling of the preVisit hook when running in parallel. Two potential fixes: 1. Add mutex locking around axe initialization in test-runner.ts: ts const mutex = new Set(); const config: TestRunnerConfig = { async preVisit(page) { while (mutex.size > 0) { await new Promise(resolve => setTimeout(resolve, 100)); } mutex.add(page); await injectAxe(page); mutex.delete(page); } }; 2. Alternative approach - move axe initialization into each test: ts const config: TestRunnerConfig = { async postRender(page, context) { await injectAxe(page); const storyContext = await getStoryContext(page, context); await configureAxe(page, { rules: storyContext.parameters?.a11y?.config?.rules, }); await checkA11y(page); } };

About Greptile

This response provides a starting point for your research, not a precise solution.

Help us improve! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

Edit Issue Bot Settings · Greptile

[akibarika]

Even though the workaround is working, I still want to identify the root cause. Currently, the test is being run manually in the Storybook UI as well.

[shilman]

We completely overhauled the a11y addon in 8.5 so it's quite possible that this is a regression. Do you have a reproduction repo you can share?

[shilman]

There's a workaround here. Looking into whether there's a better fix

chanzuckerberg/axe-storybook-testing#104

[valentinpalkovic]

Hi @akibarika,

Thank you for reporting the issue!

Can you please try out this canary release of the test runner? 0.21.1--canary.349c0dd.0. It contains a fix for your issue. Unfortunately, I can't test it because the issue is not reproducible for me.

Related PR: storybookjs/test-runner#536

[akibarika]

Hi @valentinpalkovic Thanks for the fix! Unfortunately, it still doesn’t work for me. The "page.evaluate: Error: Axe is already running." error only happens in the first two tests, but adding the following to preview.tsx works on my end:

const preview: Preview = {
  initialGlobals: {
    a11y: {
      manual: true
    }
  }
};

Due to the size and complexity of my project (it's a mix with Nx monorepo), it's quite difficult for me to create a reproducible repo for you guys. I hope this helps with troubleshooting.

[Leksat]

manual: true isn't a workaround. It just disables all automated accessibility checks. Or am I missing something?

[valentinpalkovic]

That's correct. And because Storybook doesn't run axe automatically anymore it doesn't conflict with axe which is set up via the test-runner

[andykenward]

It is a workaround when using "@storybook/test-runner": "0.21.0" and "axe-playwright": "2.0.1" to run accessibility checks. It still runs tests against the axe rules for each story when run locally and on a CI.

When running "storybook" and viewing a story in the browser. The @storybook/addon-a11y addon tab will tell you to run the checks manually. Bottom right there is a little button to run them. Which is fine for now.

Perhaps it's an issue when using chromatic?