Helix test runs are failing with "ModuleNotFoundError: No module named '_cffi_backend'"

[jkotas]

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=1354685

  import pkg_resources
Traceback (most recent call last):
  File "/tmp/helix/working/C8F90A73/p/reporter/run.py", line 13, in <module>
    from test_results_reader import read_results
  File "/private/tmp/helix/working/C8F90A73/p/reporter/test_results_reader/__init__.py", line 3, in <module>
    from helix.public import TestResult, TestResultAttachment
  File "/etc/helix/scripts/helix/public/__init__.py", line 5, in <module>
    import helix.event
  File "/etc/helix/scripts/helix/event.py", line 7, in <module>
    import helix.logs
  File "/etc/helix/scripts/helix/logs.py", line 11, in <module>
    from helix.azure_utils import get_auth_credential
  File "/etc/helix/scripts/helix/azure_utils.py", line 2, in <module>
    from azure.identity import CertificateCredential, ClientAssertionCredential, ManagedIdentityCredential, DefaultAzureCredential
  File "/etc/helix/scripts/azure/identity/__init__.py", line 10, in <module>
    from ._credentials import (
    ...<19 lines>...
    )
  File "/etc/helix/scripts/azure/identity/_credentials/__init__.py", line 5, in <module>
    from .authorization_code import AuthorizationCodeCredential
  File "/etc/helix/scripts/azure/identity/_credentials/authorization_code.py", line 9, in <module>
    from .._internal.aad_client import AadClient
  File "/etc/helix/scripts/azure/identity/_internal/__init__.py", line 5, in <module>
    from .aad_client import AadClient
  File "/etc/helix/scripts/azure/identity/_internal/aad_client.py", line 11, in <module>
    from .aad_client_base import AadClientBase
  File "/etc/helix/scripts/azure/identity/_internal/aad_client_base.py", line 21, in <module>
    from .aadclient_certificate import AadClientCertificate
  File "/etc/helix/scripts/azure/identity/_internal/aadclient_certificate.py", line 7, in <module>
    from cryptography import x509
  File "/etc/helix/scripts/cryptography/x509/__init__.py", line 7, in <module>
    from cryptography.x509 import certificate_transparency, verification
  File "/etc/helix/scripts/cryptography/x509/certificate_transparency.py", line 8, in <module>
    from cryptography.hazmat.bindings._rust import x509 as rust_x509
ModuleNotFoundError: No module named '_cffi_backend'
+ exit 1
['Microsoft.Extensions.FileProviders.Composite.Tests' END OF WORK ITEM LOG: Command exited with 1]

Build leg reported

Microsoft.Extensions.FileProviders.Composite.Tests.WorkItemExecution

Pull Request

dotnet/runtime#126191

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "ModuleNotFoundError: No module named '_cffi_backend'",
    "BuildRetry": true,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=1354685
Error message validated: [ModuleNotFoundError: No module named '_cffi_backend']
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 3/27/2026 4:07:49 PM UTC

Report

Build	Definition	Test	Pull Request
1420407	dotnet/runtime	System.Runtime.ReflectionInvokeInterpreted.Tests.WorkItemExecution	dotnet/runtime#128135
1420378	dotnet/runtime	Regression_NoOptimize_r_1.WorkItemExecution	dotnet/runtime#125436
1420365	dotnet/runtime	Methodical_ro.WorkItemExecution	dotnet/runtime#125437
1420481	dotnet/runtime	JIT_d.WorkItemExecution	dotnet/runtime#128185
1420477	dotnet/runtime	System.Runtime.Loader.RefEmitLoadContext.Tests.WorkItemExecution	dotnet/runtime#128072
1420428	dotnet/runtime	System.Net.WebProxy.Tests.WorkItemExecution	dotnet/runtime#128068
1420431	dotnet/sdk	dotnet.Tests.dll.45.WorkItemExecution	dotnet/sdk#54252
1420404	dotnet/runtime	System.Globalization.Tests.WorkItemExecution	dotnet/runtime#128049
1420293	dotnet/runtime	System.Security.Claims.Tests.WorkItemExecution	dotnet/runtime#128183
1420047	dotnet/runtime	System.Threading.ThreadPool.Tests.WorkItemExecution
1420325	dotnet/runtime	Regression_NoOptimize_r_1.WorkItemExecution	dotnet/runtime#127578
1420270	dotnet/runtime	profiler.rejit.WorkItemExecution
1420340	dotnet/runtime	Regression_do.WorkItemExecution	dotnet/runtime#128184
1420350	dotnet/runtime	System.Resources.Writer.Tests.WorkItemExecution	dotnet/runtime#128186
1420217	dotnet/runtime	JIT_ro.WorkItemExecution
1419837	dotnet/sdk	dotnet.Tests.dll.25.WorkItemExecution	dotnet/sdk#54253
1419716	dotnet/sdk	dotnet.Tests.dll.53.WorkItemExecution	dotnet/sdk#54295

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
15	17	17

[akoeplinger]

FR thread: https://teams.microsoft.com/l/message/19:afba3d1545dd45d7b79f34c1821f6055@thread.skype/1774539422238?tenantId=72f988bf-86f1-41af-91ab-2d7cd011db47&groupId=4d73664c-9f2f-450d-82a5-c2f02756606d&parentMessageId=1774539422238&teamName=.NET%20Core%20Eng%20Services%20Partners&channelName=First%20Responders&createdTime=1774539422238

[akoeplinger]

We don't see this anymore

[tannergooding]

I just got a hit on https://dev.azure.com/dnceng-public/public/_build/results?buildId=1418936&view=logs&jobId=d660e68b-c00d-5466-3901-f111ca4fe7bd&j=d660e68b-c00d-5466-3901-f111ca4fe7bd&t=cfb5bc2d-45c6-511e-7c40-507007b6e5b2&s=ff05ad62-bb9a-53b6-ce9f-72f329a63e7c

[akoeplinger]

hmm you're right, I also see a lot of results in Kusto, though only from the machine dci-mac-build-321. I wonder why Build Analysis didn't catch them.
I'll ping the FR thread.