[Linux-arm64] ModuleNotFoundError: No module named 'requests'

[EgorBo]

Widespread failures on linux-arm64

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=935169
Build error leg or test failing: Common.Tests.WorkItemExecution
Pull request: dotnet/runtime#111985

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "ModuleNotFoundError: No module named 'requests'",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=935169
Error message validated: [ModuleNotFoundError: No module named 'requests']
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/3/2025 5:46:15 PM UTC

Report

Build	Definition	Test	Pull Request
935169	dotnet/runtime	Methodical_do.WorkItemExecution	dotnet/runtime#111985

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	1	1

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Updated Helix Python code for Docker testing on Linux VMs to avoid problems with running python3 reporter/run.py after user/test code executes. Also cleaned up that code as well as our internal queue validation script.

Report

Build	Definition	Test	Pull Request
935169	dotnet/runtime	Methodical_do.WorkItemExecution	dotnet/runtime#111985

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	1

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Updated Helix Python code for Docker testing on Linux VMs to avoid problems with running python3 reporter/run.py after user/test code executes. Also cleaned up that code as well as our internal queue validation script.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

[jkotas]

Related/duplicate #4756

cc @dougbu

[dougbu]

our rollout revert should clean this problem up. I now suspect the root cause of the new problems is a permissions issue specific to Docker test scenarios

once the revert is done, we'll be heads down trying to figure out how to roll out w/o causing this problem again. @richlander I bet that'll at least temporarily leave the azurelinux.3.amd64* queues busted when used w/ most Helix Docker images

background here is I now suspect a permissions issue rather than set -e. I'm looking around to confirm my suspicions…

[richlander]

This break seems to be calling for a pre-flight/staging approach. We should be able to roll these VM updates out pretty fearlessly. That doesn't seem to be the case.

[jkotas]

our rollout revert should clean this problem up

Yes, it did.

[dougbu]

reopening to track a real and persistent fix to this issue. the revert is hopefully temporary

[dougbu]

also moved the issue b/c the problem wasn't specific to the dotnet/runtime repo

[dougbu]

transferred this back in hopes that would recreate the list of affected builds. doesn't seem to be working 😦 here at least there's one entry in the table — but it's just the original build failure

[jkotas]

transferred this back in hopes that would recreate the list of affected builds. doesn't seem to be working

You can click on "Edits" at the top post to see the past revisions where you can see the full list of affected builds.

[dougbu]

transferred this back in hopes that would recreate the list of affected builds. doesn't seem to be working

You can click on "Edits" at the top post to see the past revisions where you can see the full list of affected builds.

great thought❗taking this back to dotnet/dnceng…

[dougbu]

to avoid problems w/ expiration of Helix logs, here's a few specific errors:

by far the most common issue seen:

+ python /root/helix/work/correlation/reporter/run.py https://dev.azure.com/dnceng-public/ public 24778444 eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6Imt6UFh2cVJPMEN1UzRqU296REc4d21EM1RmcyJ9.eyJuYW1laWQiOiJjNzczZjJjMi01MTIwLTQyMDctYWZlMi1hZmFmMzVhOGJjMGEiLCJzY3AiOiJhcHBfdG9rZW4iLCJhdWkiOiI1NzdhOThkZS05N2MzLTQzOWUtOGQyOC00OWE4Nzk4NWE4M2EiLCJzaWQiOiI1YzJjYWM2NS0zOWM2LTRiM2EtYjU3Mi1hNWZiZWE2YmY5NjUiLCJCdWlsZElkIjoiY2JiMTgyNjEtYzQ4Zi00YWJiLTg2NTEtOGNkY2I1NDc0NjQ5OzkzNTY3MCIsIkRlZklkIjoiMTI5Iiwiam9icmVmIjoiZGVkMTIzMTYtOWI3My00NjE5LTkzMTMtNGMyY2E0OTdiYWM4OmYwMmI4Y2Y5LWRkNGQtNTRmYy1jMjkyLTJiYjFkMzA1YjAxOSIsInBwaWQiOiJ2c3RmczovLy9CdWlsZC9CdWlsZC85MzU2NzAiLCJvcmNoaWQiOiJkZWQxMjMxNi05YjczLTQ2MTktOTMxMy00YzJjYTQ5N2JhYzguYnVpbGQuYnVpbGRfd2FzaV93YXNtX2xpbnV4X3JlbGVhc2VfbGlicmFyeXRlc3RzX3Ntb2tlLl9fZGVmYXVsdCIsInJlcG9JZHMiOiIiLCJpc3MiOiJhcHAudnN0b2tlbi52aXN1YWxzdHVkaW8uY29tIiwiYXVkIjoiYXBwLnZzdG9rZW4udmlzdWFsc3R1ZGlvLmNvbXx2c286NmZjYzkyZTUtNzNhNy00Zjg4LThkMTMtZDkwNDViNDVmYjI3IiwibmJmIjoxNzM4Mjg4MTc5LCJleHAiOjE3MzgzMDM3Nzl9.h2II9v_m-1IW0n3opLTPybYvNnrCC2tO4mdKkAk37YsiS3x5vUSLYvx7JTmeWdWcIupmDGEDv-sSjC6I5KQFTsFpyBW8WjP1Fv07GzrbAAteQr1_qgePaz-AALjfgmXZ_T618vVIHaG6vObW9s46Dx1vfyoDBnEjFOgG6A_MAib8KADpopMPAJanqHS0srU1_9PCYzQvFyMfqYKw61DFA4C_7ba3wqekZU6QshBiQXDTr0KBLxgo6pSyYYNdo5LWxxOqKodF-1PSRImvDLR_qmulJwNU2JllBzE0UeXoDTIidLC0ueAUesoTMlSj6GJf4w3njTqU3EBJp5BCkmsk3g
Traceback (most recent call last):
  File "/root/helix/work/correlation/reporter/run.py", line 13, in <module>
    from test_results_reader import read_results
  File "/root/helix/work/correlation/reporter/test_results_reader/__init__.py", line 3, in <module>
    from helix.public import TestResult, TestResultAttachment
  File "/root/helix/scripts/helix/public/__init__.py", line 5, in <module>
    import helix.event
  File "/root/helix/scripts/helix/event.py", line 3, in <module>
    import requests
ModuleNotFoundError: No module named 'requests'

I believe I also saw errors writing to run.py.log and on chmod *.sh in the work item folder. didn't see that when checking about 15 failure logs this morning but am pretty sure

---

dotnet/runtime folks e.g., those handling XHarness these days, the following two might be longstanding problems that I just happened to notice. (I can't find where xharness-event-processor.py actually lands — it could be missing from both correlation and work item payloads for all I know)

+ python -u /root/helix/work/workitem/u/xharness-event-processor.py
python: can't open file '/root/helix/work/workitem/u/xharness-event-processor.py': [Errno 2] No such file or directory

=== TEST EXECUTION SUMMARY ===
   System.Drawing.Primitives.Tests  Total: 2439, Errors: 0, Failed: 0, Skipped: 2, Time: 1.009s
/root/helix/work/workitem/e
----- end Thu Jan 30 22:54:41 UTC 2025 ----- exit code 0 ----------------------------------------------------------
exit code 0 means Exited Successfully
dmesg: read kernel buffer failed: Operation not permitted