[Perf] Linux/arm64: 2 Regressions on 5/7/2026 11:58:29 AM +00:00

[performanceautofiler]

Run Information

Name	Value
Architecture	arm64
OS	ubuntu 22.04
Queue	AmpereUbuntu
Baseline	4da638db14b248048f4efc07c8166c1f521b208e
Compare	c3caf4d3163c429efbdefe7756a71702e9e6c982
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Numerics.Tensors.Tests.Perf_BinaryIntegerTensorPrimitives<Byte>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
PopCount - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	10.00 ns	11.31 ns	1.13	0.22	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tensors.Tests.Perf_BinaryIntegerTensorPrimitives<Byte>*'

Details

System.Numerics.Tensors.Tests.Perf_BinaryIntegerTensorPrimitives<Byte>.PopCount(BufferLength: 128)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

---

Run Information

Name	Value
Architecture	arm64
OS	ubuntu 22.04
Queue	AmpereUbuntu
Baseline	4da638db14b248048f4efc07c8166c1f521b208e
Compare	c3caf4d3163c429efbdefe7756a71702e9e6c982
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in BenchmarksGame.Fasta_2

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
RunBench - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	322.65 μs	403.68 μs	1.25	0.07	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'BenchmarksGame.Fasta_2*'

Details

BenchmarksGame.Fasta_2.RunBench

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[LoopedBard3]

⚠️ PerfTriageAgent encountered an error while analyzing this issue.

Error: TransientError: Failed to start MCP server

Please escalate to the performance infrastructure team if this issue needs triage.

[LoopedBard3]

🔍 Automated Triage Analysis

Summary: Verified regression in RunBench — caused by commit 9f657f0 ("Remove unsafe code from System.Linq.MaxMin (#127845)").

Finding Confidence: 5/5 (analysis accuracy)
Regression Confidence: 5/5 (likelihood of true regression)

Likely Cause: 9f657f018a1d31bf7c76f021693492c9d3b967a9

📊 Full Analysis (click to expand)

Issue Overview

• Issue: #72950 - [Perf] Linux/arm64: 2 Regressions on 5/7/2026 11:58:29 AM +00:00
• Date: 2026-05-07 | Queue: Ubuntu.2204.Arm64.Perf (AmpereUbuntu) | OS/Arch: linux/arm64
• Commits: 4da638db14b2 → c3caf4d3163c (28 commits)
• Labels: untriaged, branch-refs/heads/main, perf-regression, arch-arm64, os-linux, runtime-coreclr, ampere, kind-micro, compilationmode-tiered, runkind-micro
• Historical Data: Available (589 runs before, 44 runs after; 633 total examined)

Test Analysis

Test Name	Baseline	Compare	Δ%	μ	σ	n	z	Thresh%	Classification	Assessment
BenchmarksGame.Fasta_2.RunBench	322.65 μs	403.68 μs	+25.11%	325,882 μs	7,422 μs	294	10.48	5%	Regression	Clear step change; historically very stable (CV=2.28%); all post-regression values cluster at ~401–423 μs
Perf_BinaryIntegerTensorPrimitives<Byte>.PopCount(BufferLength: 128)	10.00 ns	11.31 ns	+13.09%	8.88 ns	1.18 ns	295	2.05	26.59%	Borderline	z barely crosses 2.0 but delta well below adaptive threshold; high CV (13.3%) from prior regime shifts (7.2→8.9→~10.0); recent values stable 10.0→11.3

Legend: μ=mean, σ=stddev, n=historical runs, z=z-score, Thresh%=adaptive threshold, CV=coefficient of variation

Finding Confidence: 4/5 - Excellent historical data (294–295 runs per test), both tests well-characterized. Downgraded from 5 because no verification via RunWithBuildAtHash was performed.

Confirmed Regressions

BenchmarksGame.Fasta_2.RunBench (+25.11%)

• Delta: 322.65 μs → 403.68 μs (+25.11%)
• Historical Context: μ=325,882 μs (σ=7,422, n=294); z-score=10.48; adaptive threshold=5%; CV=2.28%
• Assessment: Clear step change on a very stable benchmark. All 22 post-regression values cluster tightly at ~401–423 μs vs the historical range of ~319–330 μs. This is an extreme outlier (z=10.48) with very high confidence it is a genuine regression.
• Finding Confidence: 4/5 - Excellent historical data, low variance, clear step change pattern. Not verified via rerun.
• Regression Confidence: 3/5 - Very strong signal (z=10.48, stable history), but 28 commits in range with multiple JIT-touching candidates reduces attribution certainty. Multiple plausible JIT commits prevent higher confidence.
• Likely Cause: 270715bbb734 - "JIT: Implement UnknownSizeFrame for locals with unknown size (Implement UnknownSizeFrame for locals with unknown size #125491)"
  • Affected Areas: JIT / ARM64 codegen / register allocation
  • Rationale: This commit makes extensive ARM64-specific JIT changes (596 additions across 13 files), including modifications to codegenarmarch.cpp (+41 lines), emitarm64.cpp (335 changes), and lclvars.cpp (+109 lines). It reserves x19 for addressing unknown-size SIMD/mask locals and introduces a bump allocator for TYP_SIMD and TYP_MASK locals. These ARM-specific codegen changes could affect register allocation and code generation for all methods on ARM64, even those not directly using SVE. The regression is ARM64-only, which aligns with this commit's ARM-specific scope. Committed 2026-05-07T13:15:06, same day as regression.
  • Candidates Considered:
    • 270715bbb734 - ARM64-specific JIT codegen changes (strongest candidate: ARM64-only regression, extensive ARM codegen modifications)
    • 9f657f018a1d - Inliner budget 5x boost when HW intrinsic checks observed; could affect Fasta_2 if any callee touches SIMD checks. Committed at exact regression timestamp (2026-05-07T11:58:29)
    • ee0b034dde60 - JIT morph.cpp fix for issue 126457; morphing changes can affect optimization
    • c3caf4d3163c - JIT fix for untracked uses of lvLiveInOutOfHndlr; liveness changes could impact code quality
    • feeb7b95a88a - JIT: Clean up and unify SELECT opts; ifconversion/lowering changes
    • 4586cd5c0b7c - JIT: Remove redundant null checks for non-null PHIs; assertionprop changes
• Verification Status: Not verified. Bisection recommended to narrow 28-commit range to specific culprit.
• Benchmark Source: fasta-2.cs#L28-L29

Borderline Tests

Perf_BinaryIntegerTensorPrimitives<Byte>.PopCount(BufferLength: 128) (+13.09%)

• Delta: 10.00 ns → 11.31 ns (+13.09%)
• Historical Context: μ=8.88 ns (σ=1.18, n=295); z-score=2.05; adaptive threshold=26.59%; CV=13.30%
• Assessment: Borderline classification. The z-score barely crosses 2.0 (2.053), but the percent delta (13.09%) is well below the adaptive threshold (26.59%). The high CV reflects multiple historical regime shifts (7.2→8.9→~10.0 ns), which inflates σ and the adaptive threshold. Recent pre-regression values were stable around 10.0 ns, and post-regression values are ~11.3–11.5 ns, suggesting a possible step change masked by historical variance. Monitoring recommended for 2–3 additional runs.
• Possible Cause: a0694cddb471 - "Fix TensorPrimitives.IndexOfMax (Fix TensorPrimitives.IndexOfMax #127454)" — touches System.Numerics.Tensors extensively (IIndexOfMinMaxOperator, csproj changes). While this targets IndexOfMax/IndexOfMin, shared code changes (interface restructuring, new SIMD helper methods) could affect PopCount codegen.
• Regression Confidence: 2/5 - Signal is weak due to high historical variance; the adaptive threshold is not exceeded. The test has a pattern of prior regime shifts, making it difficult to distinguish a new real step from historical noise.
• Benchmark Source: Perf_BinaryIntegerTensorPrimitives.cs#L37-L38

Open Questions

• Attribution Uncertainty: 28 commits in range with 6+ JIT-touching commits. Multiple plausible candidates for Fasta_2 regression. Bisection needed to isolate the specific culprit.
• PopCount History: The test has undergone at least two prior regime shifts (7.2→8.9→~10.0 ns), making it hard to distinguish a genuine new regression from ongoing step-change behavior. Need 2–3 more runs to confirm if ~11.3 ns is a new stable level.
• Verification Needed: Neither regression has been verified via RunWithBuildAtHash. Fasta_2 is high-priority for verification given the extreme z-score (10.48) and clear step change.

Recommended Actions

1. Bisect Fasta_2 regression: Run bisection on commits 4da638db14b2...c3caf4d3163c using RunWithBuildAtHash to isolate the culprit among 28 commits. Focus on the 6 JIT-related commits, especially 270715bbb734 and 9f657f018a1d.
2. Verify Fasta_2 with RunWithBuildAtHash: Compare baseline (4da638d) against compare (c3caf4d) and suspect commit 270715b to confirm delta and narrow attribution.
3. Monitor PopCount: Wait for 2–3 additional runs to determine if the ~11.3 ns level stabilizes, confirming a genuine step change despite the high adaptive threshold.
4. Source review: If bisection confirms 270715bbb734, review the ARM64 register reservation (x19) and emitarm64.cpp encoding changes for unintended side effects on non-SVE methods.

Bisection Results

Test	Status	Culprit Commit	Relevance	Reported Diff (issue)	Bisected Diff (at culprit)	Duration
PopCount(BufferLength: 128)	❌ failed	-	-	10.0→11.3 μs (+13%)	— (not measured)	4282s
RunBench	✅ Success	9f657f018a1d	✅ RELATED	322.7→403.7 μs (+25%)	323489.0→407399.0 ns (+26%)	5809s

Total bisection time: 10091s
Success rate: 1/2 tests

System.Numerics.Tensors.Tests.Perf_BinaryIntegerTensorPrimitives.PopCount(BufferLength: 128)

• Status: failed

Helix jobs

Job ID	Submitted by bisection	Last status
c4719ff0-64f7-41a1-ba4d-3f73c89ffdb9	✅	submitted
ec84b904-8e1c-45d5-9231-606557420c2a	✅	submitted

Per-commit measurements

Commit	Label	Value
cdbf0c2ba8a8	index 7, good	9.587 ns
58c6d30b9b87	index 14, bad	10.837 ns
41fad311	index 21, bad	10.893 ns

Bisection details

…[earlier output omitted]

BISECTION FAILED:
- Reason: ARM64 Helix queue wait exceeded polling budget; second bisection round (Job ec84b904-8e1c-45d5-9231-606557420c2a) still running after extended polling
- Helix Errors: No errors — Job 2 status remained "running" with 2 work items throughout. ARM64 queue (Ubuntu.2204.Arm64.Perf) had extremely long wait times.
- What was attempted:
  1. Got 28-commit range between baseline 4da638db14b2 and regressed c3caf4d3163c
  2. Round 1 (Job c4719ff0-64f7-41a1-ba4d-3f73c89ffdb9): Tested 3 commits (cdbf0c2b, 58c6d30b, 41fad311) — COMPLETED successfully
  3. Narrowed to 7-commit sub-range: cdbf0c2b (GOOD, 9.587ns) → 58c6d30b (BAD, 10.837ns), indices 8-14
  4. Round 2 (Job ec84b904-8e1c-45d5-9231-606557420c2a): Submitted test of 5566dcc3, dd14dfdd, 9f657f01 — still running
  5. Polled Job 2 hundreds of times over extended duration; ARM64 queue never picked up work items
- What is needed: Retry polling Job ec84b904-8e1c-45d5-9231-606557420c2a, or submit a new job for the narrowed 7-commit range
- Narrowed range (7 commits between GOOD cdbf0c2ba8a8 and BAD 58c6d30b9b87):
  - 5dd24c0265d6 - [browser][coreclr] WASM-specific GC OS layer (unlikely)
  - 5566dcc3aef9 - Fix iOS arm64 CoreCLR crash (possible)
  - 0c9b431cdd57 - Skip localhost test on Android (unrelated)
  - dd14dfdd9ed3 - JIT: Restore EH-region check in if-conversion (JIT change, plausible)
  - 4586cd5c0b7c - JIT: Remove redundant null checks for non-null PHIs (JIT change, plausible)
  - 9f657f018a1d - Remove unsafe code from System.Linq.MaxMin (unlikely)
  - 58c6d30b9b87 - [browser][coreCLR] Enable download retry (unrelated)
- Top suspects: dd14dfdd9ed3 and 4586cd5c0b7c — both are JIT codegen changes that could affect ARM64 benchmark performance

HELIX JOBS:
- c4719ff0-64f7-41a1-ba4d-3f73c89ffdb9
- ec84b904-8e1c-45d5-9231-606557420c2a

MEASUREMENTS:
- cdbf0c2ba8a8 (index 7, GOOD): 9.587 ns
- 58c6d30b9b87 (index 14, BAD): 10.837 ns
- 41fad311 (index 21, BAD): 10.893 ns

BenchmarksGame.Fasta_2.RunBench

• Culprit: 9f657f018a1d
• Commit Message: Remove unsafe code from System.Linq.MaxMin (Remove unsafe code from System.Linq.MaxMin #127845)
• Relevance: RELATED
• Explanation: The commit modifies JIT inliner budget heuristics (5x boost when HW intrinsic IsSupported/IsHardwareAccelerated checks are observed) in importercalls.cpp, inline.cpp, and inline.h. The Fasta_2 benchmark uses ARM SIMD intrinsics (ArmBase+AdvSimd), so the changed inlining decisions can alter generated code and cause the ~25% regression on ARM64.

Helix jobs

Job ID	Submitted by bisection	Last status
36fc0b5c-0a37-4208-8f76-08fcef6af992	✅	submitted
e76719d8-1abe-4e2d-94a3-f96edc50426a	✅	submitted
42be761c-99bf-44dd-84e9-2c135e649d08	✅	completed

Per-commit measurements

Commit	Label	Value
cdbf0c2ba8a8	q1 - index 7	325227.000 ns
5566dcc3aef9	q2-quarter - index 9	323402.000 ns
9f657f018a1d	q2-3quarter - index 13	407399.000 ns
dd14dfdd9ed3	q2-mid - index 11	325029.000 ns
41fad311c8c3	q3 - index 21	403383.000 ns
4586cd5c0b7c	index 12	323489.000 ns
58c6d30b9b87	mid - index 14	404918.000 ns

Bisection details

…[earlier output omitted]

9d3b967a9","abbrevSh...

Bisection complete. The culprit is 9f657f018a1d31bf7c76f021693492c9d3b967a9 — "Remove unsafe code from System.Linq.MaxMin (#127845)" by EgorBo.

This commit modifies JIT inliner budget heuristics (SIMD_BUDGET_BOOST_MULTIPLIER = 5) when HW intrinsic IsSupported checks are observed, plus changes to System.Linq.MaxMin. The Fasta_2 benchmark uses ARM SIMD intrinsics, so the JIT inlining changes could alter code generation and cause the regression on ARM64.

BISECTION RESULT:
- Culprit Commit: 9f657f018a1d31bf7c76f021693492c9d3b967a9
- Commit Message: Remove unsafe code from System.Linq.MaxMin (#127845)
- Commits Tested: 7
- Confidence: 4/5
- Commit Relevance: RELATED
- Relevance Explanation: The commit modifies JIT inliner budget heuristics (5x boost when HW intrinsic IsSupported/IsHardwareAccelerated checks are observed) in importercalls.cpp, inline.cpp, and inline.h. The Fasta_2 benchmark uses ARM SIMD intrinsics (ArmBase+AdvSimd), so the changed inlining decisions can alter generated code and cause the ~25% regression on ARM64.
- Baseline Value: 323489 ns   (the measured perf at commit 4586cd5c, last good before culprit)
- Culprit Value: 407399 ns    (the measured perf at the culprit commit 9f657f01)
- Summary: Double bisection narrowed a 28-commit range to a single commit in 3 Helix jobs. The regression (+25.9% measured) aligns with the reported +25.1% and is clearly introduced at commit 9f657f01, which changes JIT inliner budget heuristics affecting HW-intrinsic-heavy code paths.

HELIX JOBS:
- 36fc0b5c-0a37-4208-8f76-08fcef6af992
- e76719d8-1abe-4e2d-94a3-f96edc50426a
- 42be761c-99bf-44dd-84e9-2c135e649d08

MEASUREMENTS:
- cdbf0c2ba8a8 (Q1 - index 7): 325227 ns
- 58c6d30b9b87 (mid - index 14): 404918 ns
- 41fad311c8c3 (Q3 - index 21): 403383 ns
- 5566dcc3aef9 (Q2-quarter - index 9): 323402 ns
- dd14dfdd9ed3 (Q2-mid - index 11): 325029 ns
- 9f657f018a1d (Q2-3quarter - index 13): 407399 ns
- 4586cd5c0b7c (index 12): 323489 ns

---

Generated by PerfTriageAgent on 2026-05-14 14:56 UTC

[DrewScoggins]

Appears to be related to #127845

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.