dotnet-watch tests failing

[dsplaisted]

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=629827
Build error leg or test failing: dotnet-watch.Tests.dll.1.WorkItemExecution
Pull request: #39888

Error Message

##[error].packages/microsoft.dotnet.helix.sdk/8.0.0-beta.24177.1/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(89,5): error : (NETCORE_ENGINEERING_TELEMETRY=Test) Work item dotnet-watch.Tests.dll.1 in job 82728a56-9430-4356-ab82-1c21670792a7 has failed

{
  "ErrorMessage": "",
  "ErrorPattern": "Work item dotnet-watch\\.Tests\\.dll\\.1 in job [a-z0-9\\-]+ has failed",
  "BuildRetry": true,
  "ExcludeConsoleLog": true
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=629827
Error message validated: [Work item dotnet-watch\.Tests\.dll\.1 in job [a-z0-9\-]+ has failed]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 4/4/2024 9:51:10 AM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
1144656	dotnet/sdk	Run TestBuild Tests	Log	#50737
1144505	dotnet/sdk	Run TestBuild Tests	Log	#50465
1144037	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50468
1144049	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50727
2790032	dotnet-sdk	Run TestBuild Tests	Log	#52834
1143819	dotnet/sdk	Run TestBuild Tests	Log	#50720
1143776	dotnet/sdk	Run TestBuild Tests	Log	#50699
2789962	dotnet-sdk	Run TestBuild Tests	Log	#52957
2789730	dotnet-sdk	Run TestBuild Tests	Log	#52834
1143073	dotnet/sdk	Run TestBuild Tests	Log	#50590
1142986	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50468
1142973	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50648
1142842	dotnet/sdk	Run TestBuild Tests	Log	#50692
2789052	dotnet-sdk	Run TestBuild Tests	Log	#52834
2789053	dotnet-sdk	Run TestBuild Tests	Log	#52957
1141881	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50468
1141328	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50648
1141289	dotnet/sdk	🟣 Run FullFramework Tests	Log	#50553
2787323	dotnet-sdk	Run TestBuild Tests	Log	#52834
1140255	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50657
1140098	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50648
1139975	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50648
2786176	dotnet-sdk	Run TestBuild Tests	Log	#52834
2786077	dotnet-sdk	Run FullFramework Tests	Log	#52834
1138982	dotnet/sdk	🟣 Run FullFramework Tests	Log	#50164
1138774	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50615
1137947	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50468
2785076	dotnet-sdk	Run TestBuild Tests	Log	#52834
1137781	dotnet/sdk	Run TestBuild Tests	Log	#50590
1137787	dotnet/sdk	Run TestBuild Tests	Log	#50383
1137689	dotnet/sdk	Run Tests in Helix	Log	#50583
1135299	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50554
1137275	dotnet/sdk	Run TestBuild Tests	Log	#50586
1136155	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50468
1135966	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50403
1135480	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50403
1135277	dotnet/sdk	🟣 Run FullFramework Tests	Log	#50553
1135272	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50479
1134810	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50411
2782281	dotnet-sdk	Run TestBuild Tests	Log	#52834
1134598	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50468
1134531	dotnet/sdk	🟣 Run FullFramework Tests	Log	#50479
1133800	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50518
1133807	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50468
1133743	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50518
1133629	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50518
1133302	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50479
1132773	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50508
1132701	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50468
1132491	dotnet/sdk	🟣 Run FullFramework Tests	Log	#50479
1131916	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50468
1131796	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50468
1131655	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50468
1131011	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50468
1130948	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50468
1130621	dotnet/sdk	Run FullFramework Tests	Log	#50459
1129749	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50420
1129128	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50420
2777408	dotnet-sdk	Run FullFramework Tests	Log	#52591
1129022	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50418
1128885	dotnet/sdk	Run FullFramework Tests	Log	#50415
2777326	dotnet-sdk	Run TestBuild Tests	Log	#52585
1128068	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50386
1127866	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50344
1127098	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50344
2776263	dotnet-sdk	Run TestBuild Tests	Log	#52511
2776157	dotnet-sdk	Run FullFramework Tests	Log	#52511
2776156	dotnet-sdk	Run FullFramework Tests	Log	#52510
1127290	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50371
2775965	dotnet-sdk	Run TestBuild Tests	Log	#52495
1126997	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50377
1126776	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50344
1126238	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50354
1125705	dotnet/sdk	🟣 Run FullFramework Tests	Log	#49531
1125296	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50274
1124837	dotnet/sdk	🟣 Run TestBuild Tests	Log	#49974
2773920	dotnet-sdk	Run TestBuild Tests	Log	#52387
1124711	dotnet/sdk	🟣 Run TestBuild Tests	Log	#49974
2773728	dotnet-sdk	Run TestBuild Tests	Log	#52377
1124633	dotnet/sdk	🟣 Run TestBuild Tests	Log	#49974
2773729	dotnet-sdk	Run TestBuild Tests	Log	#52383
2773398	dotnet-sdk	Run TestBuild Tests	Log	#52377
2773341	dotnet-sdk	Run TestBuild Tests	Log	#52377
1124253	dotnet/sdk	🟣 Run FullFramework Tests	Log	#49531
1124197	dotnet/sdk	🟣 Run TestBuild Tests	Log	#49974
2773175	dotnet-sdk	Run TestBuild Tests	Log	#52361
1123985	dotnet/sdk	🟣 Run TestBuild Tests	Log	#49974
1123703	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50291
1123408	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50182
1123059	dotnet/sdk	🟣 Run FullFramework Tests	Log	#49531
1122682	dotnet/sdk	🟣 Run TestBuild Tests	Log	#49974
1122642	dotnet/sdk	🟣 Run FullFramework Tests	Log	#50263
1122435	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50242
1122318	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50258
1122214	dotnet/sdk	🟣 Run TestBuild Tests	Log	#49974
2771346	dotnet-sdk	Run TestBuild Tests	Log	#52307
1121478	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50245
2771109	dotnet-sdk	🟣 Run TestBuild Tests	Log
1121255	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50168
1121248	dotnet/sdk	Run TestBuild Tests	Log	#50188
Displaying 100 of 101 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
8	25	101

[marcpopMSFT]

@tmat we still see this fairly frequently. Any luck? Here's a recent one from a codeflow PR:
https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-sdk-refs-pull-41616-merge-58511284b77c40778a/dotnet-watch.Tests.dll.1/1/console.418de7ca.log?helixlogtype=result

[tmat]

No luck yet. This might help: https://github.com/dotnet/sdk/blob/d45e40d665e647804de03d2b4eec2aae9c488cb5/test/dotnet-watch.Tests/xunit.runner.json

It will print out test info every 20 seconds a test is running.

[marcpopMSFT]

@tmat another failure this morning shows a bunch of build processes for watch apps still running... https://helixre107v0xd1eu3ibi6ka.blob.core.windows.net/dotnet-sdk-refs-pull-41958-merge-84357342ee154a5088/dotnet-watch.Tests.dll.2/1/console.3dddc463.log?helixlogtype=result

[tmat]

Looks like that's in 8.0 branch. Porting test diagnostics from 9: #41963

[Forgind]

This has been making one of my PRs fail today, so I looked into it a bit. Based on when this issue was created and which tests are failing, I suspect this PR:
#39618

I asked jasonmalinowski whether the MSBuildWorkspace is supposed to support F# because I'd found a code snippet that suggested that it shouldn't, and he said it doesn't and never did. Did that PR make us use MSBuildWorkspace in a different way than we had before?

[Forgind]

The changes to HotReloadDotNetWatcher look most suspicious to me. It looks like a substantial change to how we call MSBuild.