Integration tests on Windows pass without turning job red

[Tyriar]

https://dev.azure.com/vscode/VSCode/_build/results?buildId=155975&view=logs&j=e4714b0f-e6ff-5e04-bac1-2dcbc509f151&t=22fa355b-95c9-562c-7475-3fd3fbb94fd3

[bpasero]

@lszomoru is this something you feel familiar with?

[lszomoru]

Not really sure what is happening, but here are some findings. I have set up a topic branch in which I have introduced a test failure. Here is the pull request build from the topic branch which shows the the job fails on Linux/macOS but not on Windows.

Looking at the logs, on Linux and macOS I see the following:

1 test failed.
Extension host test runner error 1 test failed.
[32363:0614/115856.099852:INFO:CONSOLE(460)] "1 test failed.", source: vscode-file://vscode-app/mnt/vss/_work/1/s/out/vs/workbench/services/extensions/common/abstractExtensionService.js (460)
Asking native host service to exit with code 1.
[main 2025-06-14T11:58:56.107Z] Extension host with pid 32559 exited with code: 0, signal: unknown.
Exit code:   1
Failed

On Windows however, I see the following (notice the Exit code: 0)

1 test failed.
Extension host test runner error 1 test failed.
[4412:0613/143710.530:INFO:CONSOLE(460)] "1 test failed.", source: vscode-file://vscode-app/d:/a/_work/1/s/out/vs/workbench/services/extensions/common/abstractExtensionService.js (460)
Asking native host service to exit with code 1.
[main 2025-06-13T14:37:10.544Z] Extension host with pid 8240 exited with code: 0, signal: unknown.
Exit code:   0
Done

What's interesting is that the exit code on the extension host is 0, but on Linux/macOS the exit code of the script is 1 while on Windows it is 0. On Windows we are running the integration through a .bat that is run in a powershell task. I will play around with the task to see if that helps.

[lszomoru]

I have done some more testing in my topic branch, by introducing an explicit test failure in the "TypeScript tests". In the latest PR build run you can see that the task that runs integration tests has failed but the failure occured due to the failed test in "TypeScript tests" and not the failed test in "Terminal Suggest tests" even though that occurs before.

The difference between these two test suites is how they are being run:

• Terminal Suggest tests - call npm run test-extension -- -l terminal-suggest --enable-proposed-api=vscode.vscode-api-tests
• TypeScript tests - call "%INTEGRATION_TEST_ELECTRON_PATH%" %~dp0\..\extensions\typescript-language-features\test-workspace --extensionDevelopmentPath=%~dp0\..\extensions\typescript-language-features --extensionTestsPath=%~dp0\..\extensions\typescript-language-features\out\test\unit %API_TESTS_EXTRA_ARGS%

Adding a debug message confirm that call npm run test-extension -- -l terminal-suggest --enable-proposed-api=vscode.vscode-api-tests does not correctly surface the non-zero exit code and this is why the task does not fail as a result of the test failure.

I wonder if this has something to do with how call npm run works on Windows. Will dig in some more...

[bpasero]

@lszomoru good findings, the npm run test-extension I think was introduced by @connor4312 as part of supporting integration test runs with test runner extensions. Quite possible that there is runtime differences and I was actually suggesting that we should NOT be using npm run test-extension in our CI.

[lszomoru]

@lszomoru good findings, the npm run test-extension I think was introduced by @connor4312 as part of supporting integration test runs with test runner extensions. Quite possible that there is runtime differences and I was actually suggesting that we should NOT be using npm run test-extension in our CI.

@connor4312, could you please work on reverting the adoption of nom run test-extension? Thanks!