Tracking issue for CI build timeouts #76454
Comments
jkotas
added
area-Infrastructure
Known Build Error
ghost
added
the
|
Tagging subscribers to this area: @dotnet/runtime-infrastructure Issue Details {
"ErrorMessage" : "The operation was canceled.",
"BuildRetry": false
}See dotnet/arcade#11072 for more context.
|
jkotas
added
the
blocking-clean-ci
jeffschwMSFT
removed
the
untriaged
|
#82476 can be ignored, it's intentionally running tests in a slower mode to flush out failures |
|
I have noticed that the |
The log shows that the leg was waiting for 1+ hour for a Helix machine. This happens when we are sending more work to a Helix queue than what it has capacity for. Increasing the timeouts is not going to fix it. |
|
Ah, I can see it now, I've missed that. |
|
Looking at this, our queue depth is not looking great for OSX 12.
Should we be trying to spill some work to 13 and 11? cc: @dotnet/runtime-infrastructure @dotnet/dnceng |
|
The runtime time distribution seems to be consistent over the last couple weeks. We are 263043/307320 workitems sent over the last 3 days to the queue. That being said. It looks like in terms of workitem compute time, |
|
Seeing the same legs time out on as @adamsitnik, too ( I see something like this in both of the console logs: On both legs, it looks like it's the |
|
This issue seems to match also generic test timeout failures, such as is that the intent here? |
It is not the indent. I have tweaked the matching string. |
JulieLeeMSFT
removed
the
Known Build Error
|
Removing Known Build Error label because it is marking Build Analysis green. We want developers either to wait until infra timeout issue is resolved and rerun their tests or to use their judgement to use the escape path to merge. |
|
Known Build Error label is useful to get stats on the infra issues such as timeout. On the other hand, it allows PRs to get merged whle tests are not fully run. So, it is something we need to think about how we should handle such cases. |
This was the sole purpose of this tracking issue. If we do not have it marked as Known Build Error, it can be closed. |
{ "ErrorMessage" : "##[error]The operation was canceled.", "BuildRetry": false }See dotnet/arcade#11072 for more context.
Report
Summary
Known issue validation
Build: 🔎⚠️ Provided build not found. Provide a valid build in the "Build: 🔎" line.
Result validation:
Validation performed at: 11/30/2023 9:39:02 PM UTC
The text was updated successfully, but these errors were encountered: