Tests failing with Firefox 135.0

[kptdobe]

Our Firefox started failing for a couple of days. I am not sure if this is a runner issue or a playwright issue.

This is happening with Firefox 135.0 (playwright build v1475).

I have created a repo with a basic setup to illustrate the bug.

For Chrome and locally, all tests are green. Note that it only happens in the GitHub action.

As soon as there is more than one file to run test on, it fails on Firefox.

Scenario 1: https://github.com/kptdobe/web-test-runner-firefox-issue/actions/runs/13816836084

1. 2 most basic test files (one .js and one .html)
2. Success: testing files individually works fine
3. Unexpected failure: testing the 2 files together fails

Scenario 2: https://github.com/kptdobe/web-test-runner-firefox-issue/actions/runs/13816896380

1. I have incremented the browserStartTimeout as recommended - set to 2mins...
2. Success: testing files individually works fine
3. Unexpected failure: testing the 2 files together fails

Scenario 3: https://github.com/kptdobe/web-test-runner-firefox-issue/actions/runs/13816945697

1. I have added 2 more basic test files (2 .js and 2 .html)
2. Unexpected success: the test that failed before has now passed...
3. Unexpected failures: the 2 combos (2x testing JS files or 2x testing the HTML files), fails. This was to eliminate the case that maybe it is the JS+HTML test files.

Scenario 4: https://github.com/kptdobe/web-test-runner-firefox-issue/actions/runs/13817246076

1. I have incremented the timeout to 4mins
2. Success: testing files individually works fine
3. Unexpected failure: all "combo" tests failed

The fact that the test on 3.2 passed reveals that "sometimes" it works. Increasing the timeout does not solve the problem. There is something else...

[kptdobe]

Scenario 5:
https://github.com/kptdobe/web-test-runner-firefox-issue/actions/runs/13830687684 (4 mins timeout)
https://github.com/kptdobe/web-test-runner-firefox-issue/actions/runs/13830722384 (no timeout)

1. downgraded to previous versions: Firefox 134.0 via "@web/test-runner": "0.19.0" and "@playwright/test": "1.50.1"
2. in both cases, all test succeeded except the 2 HTML files

Still not sure what happens but clearly, the problem was already there before. Not sure why no one is seeing this...

[bashmish]

Can you please check if the issue remains with the configuration concurrency: 1?

[kptdobe]

Weird, I had tested it before, did not work. But now it seems to roll, all tests green: https://github.com/kptdobe/web-test-runner-firefox-issue/actions/runs/13901645165

Nice temporary workaround, thanks @bashmish.

The problem is fixed on the playwright canary version (i.e. without concurrency config)

[bashmish]

I didn't fully understand what solved it for you. Was it concurrency: 1 or new canary playwright or both?
A bit more clarity can help others who run into this problem too.

For the context: concurrency: 1 is often solving issues like this, because the issues come from the fact the modern browsers significantly slow down inactive tabs and "block" certain events in them, while WTR uses multiple tabs to parallelize the execution of individual test files.concurrency: 1 practically means "use maximum 1 tab", which then will be active.

Also, you can close this issue if solved :)

[kptdobe]

Sorry for lack of clarity:

• I have created a new test scenario and all tests are green using concurrency: 1.
• but in the test, I also have browserStartTimeout: 240000 in the config, thus it is probably the combo of the 2 configs.

Using the 2 configs is a decent workaround.

In microsoft/playwright#35183, the playwright team mentioned the problem is fixed with their latest daily built, it seems something has been changed at the Firefox level. This means that when playwright@1.52.0 is out, the workaround above will not be necessary.

I am more in favor in keeping the issue opened until it is "really" fixed. I'll test playwright@1.52.0 to confirm and then will close this issue. If someone finds this issue, a closed issue sends the wrong message because workaround still needed.

[kptdobe]

I confirm that playwright@1.52.0 fixes this issue.

[bashmish]

FYI: I've been trying to update playwright to latest among other things and set as a minimum for a long time now, but struggling with different issues. See #2879

[Skittles-69]

  Based on your scenarios, the core issue is likely resource-related concurrency within the GitHub Actions runner. To run multiple tests simultaneously, you need to manage how Playwright uses the limited resources. Here is a detailed rundown of steps you can take:

*Implement a Matrix Strategy: This is the most robust solution. Instead of running a single job that attempts to run all your test files in parallel on one runner, create a GitHub Actions matrix. The matrix will generate a separate job for each test file or a subset of files, allowing them to run on their own dedicated runners. This completely isolates the tests from each other, preventing resource contention.

*Control Concurrency with Workers: If a matrix strategy is not feasible, explicitly control the number of parallel workers Playwright uses. In your playwright.config.ts file, set the workers option to a low number (e.g., 2) or even 1 to run tests sequentially. This limits the number of concurrent browser instances, reducing the load on the runner.

*Utilize Sharding: For a large number of tests, you can use Playwright's sharding feature in combination with a matrix. The --shard command-line option divides your tests into smaller, more manageable chunks. You can then use a matrix to run each shard in a separate job. For example, npx playwright test --shard=1/2 and npx playwright test --shard=2/2 would split your tests into two groups to be run on different runners.

*Increase Runner Resources: If your tests are resource-intensive, consider using larger GitHub Actions runners. These runners have more CPU cores and memory, which can better handle parallel test execution. This is a paid option and is only available on certain GitHub plans.

*optimize Your Tests: Ensure your individual tests are efficient. Avoid unnecessary waits, use fast selectors, and implement test isolation with fixtures to prevent tests from sharing state. This reduces the overall resource demand of each test, making parallel execution more stable.