Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: isolate Electron userDataDir per-test #35235

Merged
merged 2 commits into from
Mar 17, 2025

Conversation

mxschmitt
Copy link
Member

@mxschmitt mxschmitt commented Mar 17, 2025

Taken from here. Before this patch Electron has shared the cookies across different test-runs.

const { app, protocol } = require('electron');
const path = require('path');

assert(process.argv.length > 0, 'No arguments provided. First argument needs to be a userDataDir.');
app.setPath('appData', process.argv[2]);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make sure we don't use it outside the tests that pass the dir.

@mxschmitt mxschmitt added the CQ1 label Mar 17, 2025

This comment has been minimized.

@mxschmitt mxschmitt added CQ1 and removed CQ1 labels Mar 17, 2025
@mxschmitt mxschmitt merged commit 16468e6 into microsoft:main Mar 17, 2025
89 of 106 checks passed
@mxschmitt mxschmitt deleted the isolate-electron branch March 17, 2025 18:41

This comment has been minimized.

This comment has been minimized.

Copy link
Contributor

Test results for "tests 1"

6 flaky ⚠️ [chromium-library] › tests/library/client-certificates.spec.ts:400:3 › browser › should not hang on tls errors during TLS 1.2 handshake @chromium-ubuntu-22.04-node22
⚠️ [firefox-library] › tests/library/inspector/cli-codegen-aria.spec.ts:76:7 › should update aria snapshot highlight @firefox-ubuntu-22.04-node18
⚠️ [firefox-page] › tests/page/page-evaluate.spec.ts:403:3 › should throw for too deep reference chain @firefox-ubuntu-22.04-node18
⚠️ [playwright-test] › tests/ui-mode-trace.spec.ts:341:5 › should work behind reverse proxy @macos-latest-node18-1
⚠️ [webkit-library] › tests/library/browsercontext-viewport-mobile.spec.ts:97:5 › mobile viewport › should fire orientationchange event @webkit-ubuntu-22.04-node18
⚠️ [webkit-page] › tests/page/page-screenshot.spec.ts:345:5 › page screenshot › should work while navigating @webkit-ubuntu-22.04-node18

38782 passed, 808 skipped
✔️✔️✔️

Merge workflow run.

Copy link
Contributor

Test results for "tests others"

12 fatal errors, not part of any test
4 failed
❌ [electron-page] › tests/page/page-request-continue.spec.ts:43:3 › should not allow to override unsafe HTTP headers @electron-macos-latest
❌ [electron-page] › tests/page/page-request-continue.spec.ts:43:3 › should not allow to override unsafe HTTP headers @electron-ubuntu-latest
❌ [electron-page] › tests/page/page-request-continue.spec.ts:43:3 › should not allow to override unsafe HTTP headers @electron-windows-latest
❌ [webview2] › tests/page/page-request-continue.spec.ts:43:3 › should not allow to override unsafe HTTP headers @webview2-chromium-windows

21739 passed, 512 skipped
✔️✔️✔️

Merge workflow run.

Copy link
Contributor

Test results for "tests 2"

1 fatal errors, not part of any test
17 failed
❌ [chromium-library] › tests/library/signals.spec.ts:25:5 › should close the browser when the node process closes @chrome-beta-macos-latest
❌ [chromium-library] › tests/library/browsertype-connect.spec.ts:427:5 › run-server › should reject waitForEvent before browser.onDisconnect fires @chrome-macos-latest
❌ [chromium-page] › tests/page/page-request-continue.spec.ts:43:3 › should not allow to override unsafe HTTP headers @chrome-macos-latest
❌ [chromium-page] › tests/page/page-request-continue.spec.ts:43:3 › should not allow to override unsafe HTTP headers @chrome-ubuntu-22.04
❌ [chromium-page] › tests/page/page-request-continue.spec.ts:43:3 › should not allow to override unsafe HTTP headers @chrome-windows-latest
❌ [chromium-library] › tests/library/inspector/cli-codegen-2.spec.ts:454:7 › cli codegen › should save assets via SIGINT @msedge-dev-macos-latest
❌ [chromium-library] › tests/library/inspector/cli-codegen-test.spec.ts:112:5 › should generate routeFromHAR with --save-har and --save-har-glob @msedge-dev-macos-latest
❌ [chromium-library] › tests/library/proxy.spec.ts:93:11 › should proxy local network requests › with other bypasses › localhost @msedge-dev-macos-latest
❌ [chromium-library] › tests/library/proxy.spec.ts:93:11 › should proxy local network requests › with other bypasses › loopback address @msedge-dev-macos-latest
❌ [chromium-library] › tests/library/browsertype-connect.spec.ts:910:7 › run-server › socks proxy › should proxy local.playwright requests @msedge-macos-latest
❌ [chromium-library] › tests/library/browsertype-launch-server.spec.ts:43:5 › launch server › should work with wsPath @msedge-macos-latest
❌ [chromium-library] › tests/library/inspector/cli-codegen-2.spec.ts:454:7 › cli codegen › should save assets via SIGINT @msedge-macos-latest
❌ [chromium-page] › tests/page/page-request-continue.spec.ts:43:3 › should not allow to override unsafe HTTP headers @msedge-macos-latest
❌ [chromium-page] › tests/page/page-request-continue.spec.ts:43:3 › should not allow to override unsafe HTTP headers @msedge-ubuntu-22.04
❌ [chromium-page] › tests/page/page-request-continue.spec.ts:43:3 › should not allow to override unsafe HTTP headers @msedge-windows-latest
❌ [webkit-library] › tests/library/tracing.spec.ts:432:14 › should produce screencast frames scale @webkit-macos-14-xlarge
❌ [webkit-library] › tests/library/tracing.spec.ts:432:14 › should produce screencast frames scale @webkit-macos-15-xlarge

106 flaky ⚠️ [chromium-library] › tests/library/beforeunload.spec.ts:20:3 › should close browser with beforeunload page @channel-chromium-macos-latest
⚠️ [chromium-library] › tests/library/browser.spec.ts:54:5 › should dispatch page.on(close) upon browser.close and reject evaluate @channel-chromium-macos-latest
⚠️ [chromium-library] › tests/library/browsercontext-base-url.spec.ts:37:3 › should construct a new URL when a baseURL in browserType.launchPersistentContext is passed to page.goto @channel-chromium-macos-latest
⚠️ [chromium-library] › tests/library/browsercontext-fetch.spec.ts:1231:3 › should work with connectOverCDP @channel-chromium-macos-latest
⚠️ [chromium-library] › tests/library/browsercontext-proxy.spec.ts:27:3 › should work when passing the proxy only on the context level @channel-chromium-macos-latest
⚠️ [chromium-page] › tests/page/locator-frame.spec.ts:272:3 › should work with COEP/COOP/CORP isolated iframe @channel-chromium-macos-latest
⚠️ [chromium-library] › tests/library/browser.spec.ts:54:5 › should dispatch page.on(close) upon browser.close and reject evaluate @chrome-beta-macos-latest
⚠️ [chromium-library] › tests/library/browsertype-launch.spec.ts:115:3 › should fire close event for all contexts @chrome-beta-macos-latest
⚠️ [chromium-library] › tests/library/fetch-proxy.spec.ts:21:3 › context request should pick up proxy credentials @chrome-beta-macos-latest
⚠️ [chromium-library] › tests/library/logger.spec.ts:34:3 › should log context-level @chrome-beta-macos-latest
⚠️ [chromium-library] › tests/library/browsercontext-fetch.spec.ts:1231:3 › should work with connectOverCDP @chrome-beta-windows-latest
⚠️ [chromium-library] › tests/library/browsercontext-proxy.spec.ts:27:3 › should work when passing the proxy only on the context level @chrome-beta-windows-latest
⚠️ [chromium-library] › tests/library/video.spec.ts:580:5 › screencast › should capture static page in persistent context @smoke @chrome-beta-windows-latest
⚠️ [chromium-library] › tests/library/browser.spec.ts:54:5 › should dispatch page.on(close) upon browser.close and reject evaluate @chrome-macos-latest
⚠️ [chromium-library] › tests/library/inspector/cli-codegen-2.spec.ts:454:7 › cli codegen › should save assets via SIGINT @chrome-macos-latest
⚠️ [chromium-library] › tests/library/inspector/cli-codegen-pytest.spec.ts:57:5 › should save the codegen output to a file if specified @chrome-macos-latest
⚠️ [chromium-library] › tests/library/inspector/cli-codegen-python-async.spec.ts:82:5 › should save the codegen output to a file if specified @chrome-macos-latest
⚠️ [chromium-library] › tests/library/inspector/cli-codegen-1.spec.ts:116:7 › cli codegen › should click after same-document navigation @chromium-headed-ubuntu-24.04
⚠️ [chromium-library] › tests/library/trace-viewer.spec.ts:1298:1 › should pick locator in iframe @chromium-macos-13-large
⚠️ [chromium-library] › tests/library/chromium/oopif.spec.ts:284:3 › should click @chromium-tip-of-tree-headless-shell-ubuntu-22.04
⚠️ [chromium-library] › tests/library/inspector/cli-codegen-python.spec.ts:133:5 › should work with --save-har @chromium-tip-of-tree-ubuntu-22.04--headed
⚠️ [chromium-page] › tests/page/page-request-continue.spec.ts:67:3 › should delete header with undefined value @chromium-windows-latest
⚠️ [firefox-library] › tests/library/video.spec.ts:163:5 › screencast › should work with old options @firefox-beta-macos-latest
⚠️ [firefox-page] › tests/page/page-evaluate.spec.ts:403:3 › should throw for too deep reference chain @firefox-beta-ubuntu-22.04
⚠️ [firefox-library] › tests/library/browsercontext-storage-state.spec.ts:359:3 › should support IndexedDB @firefox-beta-windows-latest
⚠️ [firefox-page] › tests/page/page-evaluate.spec.ts:403:3 › should throw for too deep reference chain @firefox-beta-windows-latest
⚠️ [firefox-library] › tests/library/capabilities.spec.ts:252:3 › requestFullscreen @firefox-headed-macos-14-xlarge
⚠️ [firefox-library] › tests/library/inspector/cli-codegen-aria.spec.ts:76:7 › should update aria snapshot highlight @firefox-headed-macos-14-xlarge
⚠️ [firefox-library] › tests/library/inspector/cli-codegen-1.spec.ts:891:7 › cli codegen › should record slider @firefox-headed-ubuntu-24.04
⚠️ [firefox-page] › tests/page/page-add-locator-handler.spec.ts:230:5 › should work when owner frame detaches @firefox-headed-ubuntu-24.04
⚠️ [firefox-page] › tests/page/page-evaluate.spec.ts:403:3 › should throw for too deep reference chain @firefox-headed-ubuntu-24.04
⚠️ [firefox-library] › tests/library/browsercontext-locale.spec.ts:141:3 › should not change default locale in another context @firefox-headed-windows-latest
⚠️ [firefox-library] › tests/library/hit-target.spec.ts:398:3 › should click in iframe with padding @firefox-headed-windows-latest
⚠️ [firefox-library] › tests/library/tracing.spec.ts:263:5 › should not include trace resources from the previous chunks @firefox-headed-windows-latest
⚠️ [firefox-page] › tests/page/page-click-timeout-4.spec.ts:20:3 › should timeout waiting for stable position @firefox-headed-windows-latest
⚠️ [firefox-page] › tests/page/page-evaluate.spec.ts:403:3 › should throw for too deep reference chain @firefox-headed-windows-latest
⚠️ [firefox-page] › tests/page/page-mouse.spec.ts:174:3 › should select the text with mouse @firefox-headed-windows-latest
⚠️ [firefox-library] › tests/library/trace-viewer.spec.ts:1298:1 › should pick locator in iframe @firefox-macos-13-large
⚠️ [firefox-library] › tests/library/video.spec.ts:163:5 › screencast › should work with old options @firefox-macos-13-large
⚠️ [firefox-page] › tests/page/page-evaluate.spec.ts:403:3 › should throw for too deep reference chain @firefox-macos-13-large
⚠️ [firefox-library] › tests/library/video.spec.ts:163:5 › screencast › should work with old options @firefox-macos-13-xlarge
⚠️ [firefox-library] › tests/library/video.spec.ts:163:5 › screencast › should work with old options @firefox-macos-14-large
⚠️ [firefox-page] › tests/page/page-evaluate.spec.ts:403:3 › should throw for too deep reference chain @firefox-macos-14-large
⚠️ [firefox-library] › tests/library/video.spec.ts:163:5 › screencast › should work with old options @firefox-macos-14-xlarge
⚠️ [firefox-page] › tests/page/page-evaluate.spec.ts:403:3 › should throw for too deep reference chain @firefox-ubuntu-24.04
⚠️ [firefox-page] › tests/page/page-evaluate.spec.ts:403:3 › should throw for too deep reference chain @firefox-windows-latest
⚠️ [chromium-library] › tests/library/beforeunload.spec.ts:20:3 › should close browser with beforeunload page @msedge-beta-macos-latest
⚠️ [chromium-library] › tests/library/browser.spec.ts:54:5 › should dispatch page.on(close) upon browser.close and reject evaluate @msedge-beta-macos-latest
⚠️ [chromium-library] › tests/library/browsercontext-base-url.spec.ts:37:3 › should construct a new URL when a baseURL in browserType.launchPersistentContext is passed to page.goto @msedge-beta-macos-latest
⚠️ [chromium-library] › tests/library/browsercontext-reuse.spec.ts:30:1 › should re-add binding after reset @msedge-beta-macos-latest
⚠️ [chromium-library] › tests/library/browsertype-connect.spec.ts:188:5 › launchServer › should be able to visit ipv6 through localhost @msedge-beta-macos-latest
⚠️ [chromium-library] › tests/library/headful.spec.ts:114:3 › should close browser after context menu was triggered @msedge-beta-macos-latest
⚠️ [chromium-library] › tests/library/inspector/cli-codegen-csharp.spec.ts:171:5 › should work with --save-har @msedge-beta-macos-latest
⚠️ [chromium-library] › tests/library/browsercontext-reuse.spec.ts:30:1 › should re-add binding after reset @msedge-beta-ubuntu-22.04
⚠️ [chromium-library] › tests/library/browsercontext-reuse.spec.ts:30:1 › should re-add binding after reset @msedge-beta-windows-latest
⚠️ [chromium-library] › tests/library/trace-viewer.spec.ts:1744:1 › should toggle canvas rendering @msedge-beta-windows-latest
⚠️ [chromium-library] › tests/library/browsercontext-base-url.spec.ts:37:3 › should construct a new URL when a baseURL in browserType.launchPersistentContext is passed to page.goto @msedge-dev-macos-latest
⚠️ [chromium-library] › tests/library/browsercontext-fetch.spec.ts:1231:3 › should work with connectOverCDP @msedge-dev-macos-latest
⚠️ [chromium-library] › tests/library/browsertype-connect.spec.ts:80:5 › launchServer › should connect over wss @msedge-dev-macos-latest
⚠️ [chromium-library] › tests/library/browsertype-launch-server.spec.ts:23:5 › launch server › should work @msedge-dev-macos-latest
⚠️ [chromium-library] › tests/library/downloads-path.spec.ts:32:5 › downloads path › should keep downloadsPath folder @msedge-dev-macos-latest
⚠️ [chromium-library] › tests/library/downloads-path.spec.ts:91:5 › downloads path › should accept downloads in persistent context @msedge-dev-macos-latest
⚠️ [chromium-library] › tests/library/inspector/cli-codegen-aria.spec.ts:76:7 › should update aria snapshot highlight @msedge-dev-macos-latest
⚠️ [chromium-library] › tests/library/inspector/cli-codegen-csharp.spec.ts:171:5 › should work with --save-har @msedge-dev-macos-latest
⚠️ [chromium-library] › tests/library/inspector/cli-codegen-test.spec.ts:88:5 › should not generate recordHAR with --save-har @msedge-dev-macos-latest
⚠️ [chromium-library] › tests/library/browsercontext-reuse.spec.ts:30:1 › should re-add binding after reset @msedge-dev-ubuntu-22.04
⚠️ [chromium-library] › tests/library/browsercontext-base-url.spec.ts:37:3 › should construct a new URL when a baseURL in browserType.launchPersistentContext is passed to page.goto @msedge-macos-latest
⚠️ [chromium-library] › tests/library/browsercontext-proxy.spec.ts:27:3 › should work when passing the proxy only on the context level @msedge-macos-latest
⚠️ [chromium-library] › tests/library/browsercontext-reuse.spec.ts:30:1 › should re-add binding after reset @msedge-macos-latest
⚠️ [chromium-library] › tests/library/browsertype-connect.spec.ts:173:5 › launchServer › should ignore page.pause when headed @msedge-macos-latest
⚠️ [chromium-library] › tests/library/browsertype-connect.spec.ts:925:7 › run-server › socks proxy › should lead to the error page for forwarded requests when the connection is refused @msedge-macos-latest
⚠️ [chromium-library] › tests/library/browsertype-launch-server.spec.ts:50:5 › launch server › should work when wsPath is missing leading slash @msedge-macos-latest
⚠️ [chromium-library] › tests/library/inspector/cli-codegen-pick-locator.spec.ts:35:7 › should update locator highlight @msedge-macos-latest
⚠️ [chromium-library] › tests/library/browsercontext-reuse.spec.ts:30:1 › should re-add binding after reset @msedge-ubuntu-22.04
⚠️ [chromium-library] › tests/library/browsercontext-proxy.spec.ts:153:3 › should use proxy twice @msedge-windows-latest
⚠️ [chromium-library] › tests/library/browsercontext-proxy.spec.ts:264:3 › should isolate proxy credentials between contexts @msedge-windows-latest
⚠️ [chromium-library] › tests/library/browsercontext-proxy.spec.ts:294:3 › should exclude patterns @msedge-windows-latest
⚠️ [chromium-library] › tests/library/browsercontext-reuse.spec.ts:30:1 › should re-add binding after reset @msedge-windows-latest
⚠️ [chromium-library] › tests/library/video.spec.ts:379:5 › screencast › should capture navigation @msedge-windows-latest
⚠️ [chromium-library] › tests/library/headful.spec.ts:233:3 › should click in OOPIF @tracing-chromium-tip-of-tree
⚠️ [firefox-page] › tests/page/page-evaluate.spec.ts:403:3 › should throw for too deep reference chain @tracing-firefox
⚠️ [webkit-library] › tests/library/screenshot.spec.ts:205:14 › element screenshot › element screenshot should work with a mobile viewport @tracing-webkit
⚠️ [webkit-library] › tests/library/video.spec.ts:411:5 › screencast › should capture css transformation @tracing-webkit
⚠️ [webkit-page] › tests/page/page-event-popup.spec.ts:28:3 › should work with window features @tracing-webkit
⚠️ [webkit-page] › tests/page/page-screenshot.spec.ts:345:5 › page screenshot › should work while navigating @tracing-webkit
⚠️ [webkit-library] › tests/library/inspector/cli-codegen-1.spec.ts:610:7 › cli codegen › should select @webkit-headed-macos-14-xlarge
⚠️ [webkit-library] › tests/library/inspector/cli-codegen-pick-locator.spec.ts:35:7 › should update locator highlight @webkit-headed-macos-14-xlarge
⚠️ [webkit-library] › tests/library/inspector/cli-codegen-1.spec.ts:381:7 › cli codegen › should press @webkit-headed-ubuntu-24.04
⚠️ [webkit-page] › tests/page/page-click.spec.ts:261:3 › should click on checkbox input and toggle @webkit-headed-ubuntu-24.04
⚠️ [webkit-library] › tests/library/inspector/cli-codegen-1.spec.ts:96:7 › cli codegen › should ignore programmatic events @webkit-headed-windows-latest
⚠️ [webkit-library] › tests/library/inspector/cli-codegen-1.spec.ts:540:7 › cli codegen › should check a radio button @webkit-headed-windows-latest
⚠️ [webkit-library] › tests/library/inspector/cli-codegen-1.spec.ts:610:7 › cli codegen › should select @webkit-headed-windows-latest
⚠️ [webkit-library] › tests/library/inspector/cli-codegen-1.spec.ts:684:7 › cli codegen › should await popup @webkit-headed-windows-latest
⚠️ [webkit-library] › tests/library/inspector/cli-codegen-3.spec.ts:520:7 › cli codegen › should generate getByPlaceholder @webkit-headed-windows-latest
⚠️ [webkit-library] › tests/library/inspector/cli-codegen-3.spec.ts:578:7 › cli codegen › should generate getByLabel @webkit-headed-windows-latest
⚠️ [webkit-library] › tests/library/inspector/cli-codegen-3.spec.ts:737:7 › cli codegen › should assert value @webkit-headed-windows-latest
⚠️ [webkit-library] › tests/library/inspector/cli-codegen-3.spec.ts:809:7 › cli codegen › should assert value on disabled select @webkit-headed-windows-latest
⚠️ [webkit-library] › tests/library/video.spec.ts:379:5 › screencast › should capture navigation @webkit-macos-14-large
⚠️ [webkit-library] › tests/library/tracing.spec.ts:432:14 › should produce screencast frames crop @webkit-macos-14-xlarge
⚠️ [webkit-page] › tests/page/page-request-fallback.spec.ts:263:5 › post data › should amend json post data @webkit-macos-14-xlarge
⚠️ [webkit-page] › tests/page/wheel.spec.ts:70:3 › should dispatch wheel events after context menu was opened @webkit-macos-14-xlarge
⚠️ [webkit-page] › tests/page/wheel.spec.ts:166:3 › should set the modifiers @webkit-macos-14-xlarge
⚠️ [webkit-library] › tests/library/tracing.spec.ts:432:14 › should produce screencast frames crop @webkit-macos-15-xlarge
⚠️ [webkit-page] › tests/page/page-request-continue.spec.ts:267:5 › post data › should amend longer post data @webkit-macos-15-xlarge
⚠️ [webkit-page] › tests/page/wheel.spec.ts:185:3 › should scroll horizontally @webkit-macos-15-xlarge
⚠️ [webkit-library] › tests/library/trace-viewer.spec.ts:1298:1 › should pick locator in iframe @webkit-ubuntu-24.04

236484 passed, 9285 skipped
✔️✔️✔️

Merge workflow run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants