[Perf] Linux/arm64: 2 Regressions on 5/11/2026 8:05:21 PM +00:00

[performanceautofiler]

Run Information

Name	Value
Architecture	arm64
OS	azurelinux 3.0
Queue	CobaltAzureLinux
Baseline	0bccdce838fcce449299d5889ebd5a2a94eb3d28
Compare	983639bd7eb45543ee98e6bc759b728238945d1d
Diff	Diff
Configs	CompilationMode:tiered, R2RType:r2r_interpreter, RunKind:micro

Regressions in System.Threading.Channels.Tests.BoundedChannelPerfTests

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
ReadAsyncThenWriteAsync - Duration of single invocation ADX - Test Multi Config Graph	408.81 ns	500.53 ns	1.22	0.12	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Threading.Channels.Tests.BoundedChannelPerfTests*'

Details

System.Threading.Channels.Tests.BoundedChannelPerfTests.ReadAsyncThenWriteAsync

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

---

Run Information

Name	Value
Architecture	arm64
OS	azurelinux 3.0
Queue	CobaltAzureLinux
Baseline	0bccdce838fcce449299d5889ebd5a2a94eb3d28
Compare	3caa97ab2ceaf0185ca6de09c6cc497dccfe6f3e
Diff	Diff
Configs	CompilationMode:tiered, R2RType:r2r_interpreter, RunKind:micro

Regressions in Microsoft.AspNetCore.Server.Kestrel.Performance.PipeThroughputBenchmark

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
Parse_ParallelAsync - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	389.61 ns	435.69 ns	1.12	0.29	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Microsoft.AspNetCore.Server.Kestrel.Performance.PipeThroughputBenchmark*'

Details

Microsoft.AspNetCore.Server.Kestrel.Performance.PipeThroughputBenchmark.Parse_ParallelAsync(Length: 128, Chunks: 1)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[LoopedBard3]

🔍 Automated Triage Analysis

Summary: 1. Verify regression: Run RunWithBuildAtHash comparing baseline 0bccdce838fc against suspected commit 4bdd9f848863 for both System.Threading.Channels.Tests.BoundedChannelPerfTests.ReadAsyncThenWriteAsync and `Microsoft.AspNetCore.Server.Kestrel.Performance.PipeThroughputBenchmark.Parse_ParallelAsync(Length: 128, Chunks: Bisection skipped — autofile run configuration is not supported by the bisection harness.

Finding Confidence: 4/5 (analysis accuracy)
Regression Confidence: 4/5 (likelihood of true regression)

📊 Full Analysis (click to expand)

Issue Overview

• Issue: #72940 - [Perf] Linux/arm64: 2 Regressions on 5/11/2026 8:05:21 PM +00:00
• Date: 2026-05-11 | Queue: AzureLinux.3.Cobalt.Arm64.Perf | OS/Arch: linux/arm64
• Commits: 0bccdce838fc → 983639bd7eb4 (14 commits)
• Labels: untriaged, perf-regression, os-linux, arch-arm64, runtime-coreclr, kind-micro, compilationmode-tiered, runkind-micro, r2rtype-r2r_interpreter
• Historical Data: Available (90 prior runs per test, 205 total runs examined)

Test Analysis

Test Name	Baseline	Compare	Δ%	μ	σ	n	z	Thresh%	Noise?	Assessment
BoundedChannelPerfTests.ReadAsyncThenWriteAsync	408.81 ns	500.53 ns	+22.44%	412.08	31.39	90	2.82	15.23%	N	Step change, clear regression
PipeThroughputBenchmark.Parse_ParallelAsync(Length: 128, Chunks: 1)	389.61 ns	435.69 ns	+11.83%	391.15	20.65	90	2.16	10.56%	N	Drift pattern, pre-existing upward trend

Legend: μ=mean, σ=stddev, n=runs, z=z-score, Thresh%=adaptive threshold

Finding Confidence: 4/5 - Strong historical data (90 runs per test), sufficient after-window coverage (12-13 runs). Both tests have statistically significant deviations (z ≥ 2.0). Minor downgrade due to moderate CV (7.6% Channels, 5.3% Kestrel) and some low outliers inflating Channels σ.

Confirmed Regressions

System.Threading.Channels.Tests.BoundedChannelPerfTests.ReadAsyncThenWriteAsync (+22.44%)

• Delta: 408.81 ns → 500.53 ns (+22.44%)
• Historical Context: μ=412.08 (σ=31.39, n=90); z-score=2.82; adaptive threshold=15.23%
• Assessment: Step change — after-window shows first 4 runs near historical baseline (~411-417 ns), then a clear sustained step up to ~483-500 ns. This indicates a discrete code change introduced during the regression window.
• Test Source: ReadAsyncThenWriteAsync — async channel read/write benchmark using ValueTask<int> read followed by WriteAsync.
• Finding Confidence: 4/5 - Strong historical data, clear step change pattern, z=2.82 well above threshold
• Regression Confidence: 4/5 - Direct interpreter async code path hit, strong signal (z=2.82), small commit range (14 commits), surgical change in async suspend/resume logic
• Likely Cause: 4bdd9f848863 - "Move the return var out of liveLocalsIntervals (Move the return var out of liveLocalsIntervals #127931)"
  • Affected Areas: CoreCLR Interpreter / Async Suspend-Resume
  • Rationale: This commit directly modifies the interpreter's async suspend and resume path — exactly the code path exercised by this async channel benchmark running under r2r_interpreter mode. The change introduces additional work during async suspension: it separates the return value from the live locals set, adds a new returnValueContinuationDataSize field to InterpAsyncSuspendData, shifts the continuation data start offset (GetResultStorage() + returnValueContinuationDataSize), and adds an explicit memcpy for the return value on the resume path. This extra memory copy and offset computation on every async suspend/resume cycle would directly increase per-invocation overhead for ReadAsyncThenWriteAsync, which performs a ValueTask<int> read and WriteAsync on each iteration. The step change pattern in the after-window aligns with builds incorporating this commit.
  • Candidates Considered:
    • 4bdd9f848863 - Primary: Direct interpreter async suspend/resume changes, adds memcpy overhead
    • 0306ba79b0cf - JIT assertion propagation enhancement; could theoretically affect codegen but is an optimization (should help, not hurt)
    • 553a1fe1e959 - Arm64 SVE2 intrinsics addition; unrelated to async channel operations
• Verification Status: Not verified — verification via RunWithBuildAtHash recommended to confirm

Microsoft.AspNetCore.Server.Kestrel.Performance.PipeThroughputBenchmark.Parse_ParallelAsync(Length: 128, Chunks: 1) (+11.83%)

• Delta: 389.61 ns → 435.69 ns (+11.83%)
• Historical Context: μ=391.15 (σ=20.65, n=90); z-score=2.16; adaptive threshold=10.56%
• Assessment: Drift pattern — recentBeforeMean (404.1) was already trending above earlyBeforeMean (388.3), and a 445.1 ns outlier appears in the before-window tail. After-values are consistently elevated (391-452 ns) with one low outlier at 391.3 ns. The signal is real but partially attributable to a pre-existing upward drift.
• Test Source: Parse_ParallelAsync — parallel pipe reader/writer benchmark using Task.Run with async reader and writer.
• Finding Confidence: 3/5 - Good historical data but pre-existing drift complicates attribution; CV=5.3% is moderate
• Regression Confidence: 3/5 - Signal exceeds threshold (z=2.16) but drift pattern suggests partial pre-existing trend; same interpreter async cause is plausible but less certain
• Likely Cause: 4bdd9f848863 - "Move the return var out of liveLocalsIntervals (Move the return var out of liveLocalsIntervals #127931)"
  • Affected Areas: CoreCLR Interpreter / Async Suspend-Resume
  • Rationale: Same interpreter async suspend/resume changes as the Channels test. Parse_ParallelAsync uses Task.Run with parallel async reader/writer, exercising async continuation paths in the interpreter. The additional memcpy and offset computation overhead would also impact this test. However, confidence is lower because the pre-existing upward drift in before-values suggests other factors may also be contributing (potentially incremental arm64-specific changes or scheduler/IO changes accumulating over recent commits).
  • Candidates Considered:
    • 4bdd9f848863 - Primary: Same interpreter async changes affecting parallel async workloads
    • 15aceeac0dcc - Replace GetOsPageSize() with minipal_getpagesize() across CoreCLR; unlikely but could theoretically affect memory allocation patterns
    • 0306ba79b0cf - JIT branch folding optimization; should be neutral or positive
• Verification Status: Not verified — verification recommended, especially given drift pattern

Open Questions

• Historical Data: Available with strong coverage (90 runs per test). Some extreme low outliers in Channels beforeValues (257.5, 260.5, 261.2 ns) inflate historicalStdDev and raise the adaptive threshold; classification remains Regression even with the elevated threshold.
• Confidence Limitations: Pre-existing drift in Kestrel test complicates attribution. The Channels test step change is cleaner and more attributable.
• Verification Needed: RunWithBuildAtHash verification on commits 0bccdce838fc (baseline) and 4bdd9f848863 (suspected cause) would confirm whether the interpreter async change is the culprit for both tests.
• Bisection Status: Not performed — 14 commits is a manageable range; direct verification on the suspected commit should be sufficient.

Recommended Actions

1. Verify regression: Run RunWithBuildAtHash comparing baseline 0bccdce838fc against suspected commit 4bdd9f848863 for both System.Threading.Channels.Tests.BoundedChannelPerfTests.ReadAsyncThenWriteAsync and Microsoft.AspNetCore.Server.Kestrel.Performance.PipeThroughputBenchmark.Parse_ParallelAsync(Length: 128, Chunks: 1).
2. Source review: Review 4bdd9f848863 focusing on whether the additional memcpy for returnValueContinuationDataSize on every async resume can be optimized — potentially by deferring the copy or reducing the alignment overhead.
3. Investigate Kestrel drift: Examine the Kestrel test's upward trend in recent before-values to determine if there is a secondary cause beyond the interpreter change (e.g., incremental arm64 scheduler changes).

---

Generated by PerfTriageAgent on 2026-05-14 08:43 UTC