Keep measured matrix out of TUnit and gate benchmark quality

Author: KSemenenko

.github/workflows/performance.yml

@@ -109,9 +109,10 @@ jobs:
         run: |
           dotnet experiments/Synapse.ReferenceBenchmarks/bin/Release/net10.0/Synapse.ReferenceBenchmarks.dll report \
             --input "${RUNNER_TEMP}/synapse-benchmark.json" \
-            --summary "${GITHUB_STEP_SUMMARY}"
+            --summary "${GITHUB_STEP_SUMMARY}" --require-quality
 
       - name: Preserve raw per-round evidence
+        if: always()
         uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
         with:
           name: performance-${{ matrix.runtime_identifier }}

docs/ADR/ADR-005-benchmark-timing-provenance.md

@@ -36,10 +36,12 @@ shared libraries, runtime heaps, and driver memory make that subtraction
 invalid. A missing platform metric remains null with its scope recorded.
 
 Functional CI and performance evidence are separate workflows. Functional
-tests may make short real-process quality and schema assertions but do not
-assert throughput or use long generation as a performance proxy. The manually
+TUnit checks can validate rendering against recorded raw evidence, but do not
+run the measured matrix or assert throughput. The manually
 dispatched performance workflow runs its own macOS/Linux/Windows matrix and
-retains every measured round as an artifact with a descriptive report. Hosted
+retains every measured round as an artifact with a descriptive report. An
+output mismatch fails the job after writing the report, while the raw artifact
+is still uploaded. Hosted
 runners are variable hardware, so their results are never pooled across OSes
 or compared to the owner's Mac as a release winner verdict.
 

docs/Features/Benchmarking.md

@@ -57,18 +57,22 @@ the 30-pair randomized release benchmark, and 8-token decode timing remains
 too short for an optimization win claim.
 
 GitHub Actions keeps correctness and performance separate. `verify.yml` runs
-the functional suite on macOS ARM64, Ubuntu x64, and Windows x64; its matrix
-smoke checks short-output quality and schema, not speed thresholds.
+the functional suite on macOS ARM64, Ubuntu x64, and Windows x64; it does not
+run the measured matrix or enforce speed thresholds. Existing recorded JSON
+tests report formatting and mismatch rejection without running performance
+samples inside TUnit.
 `performance.yml` is a separate manually dispatched three-OS CPU matrix. It
 fetches only the pinned Qwen smoke GGUF, builds the pinned competitors, runs
 three warm-ups plus five measured fresh-process rounds with a two-thread cap,
 and uploads unmodified per-round JSON for each runner. The C# `report --input
-<raw.json> [--summary <path>]` command renders workload, model digest,
+<raw.json> [--summary <path>] [--require-quality]` command renders workload, model digest,
 platform, quality, median process wall time, peak RSS, Mac footprint, and
-reported decode phase into the GitHub job summary. Each runner is a distinct
+reported decode phase into the GitHub job summary. The quality flag fails a
+job when output diverges, while the raw artifact is retained. Each runner is a distinct
 cohort; this eight-token diagnostic neither replaces long generation nor
-passes the 30-pair release benchmark gate. A run has not been verified until
-its actual Actions job and artifact succeed.
+passes the 30-pair release benchmark gate. The first actual Actions run on
+`9567d06` passed all three performance jobs and published three raw artifacts;
+the quality-gate CLI change still requires a fresh workflow run.
 
 Two 10-turn modes use `benchmarks/scenarios/travel-planner-10-turns.json`:
 

experiments/Synapse.ReferenceBenchmarks/DiagnosticMatrixReportCommand.cs

@@ -9,10 +9,13 @@ internal static class DiagnosticMatrixReportCommand
 
     public static async Task<int> RunAsync(string[] arguments)
     {
-        if (arguments.Length is not (2 or 4) || arguments[0] != "--input" ||
-            (arguments.Length == 4 && arguments[2] != "--summary"))
+        var requireQuality = arguments.LastOrDefault() == "--require-quality";
+        var optionLength = arguments.Length - (requireQuality ? 1 : 0);
+        if (optionLength is not (2 or 4) || arguments[0] != "--input" ||
+            (optionLength == 4 && arguments[2] != "--summary"))
         {
-            Console.Error.WriteLine("Usage: report --input <matrix.json> [--summary <github-step-summary>]");
+            Console.Error.WriteLine("Usage: report --input <matrix.json> " +
+                "[--summary <github-step-summary>] [--require-quality]");
             return 2;
         }
 
@@ -22,11 +25,18 @@ public static async Task<int> RunAsync(string[] arguments)
             using var document = await JsonDocument.ParseAsync(stream).ConfigureAwait(false);
             var report = Render(document.RootElement);
             Console.Write(report);
-            if (arguments.Length == 4)
+            if (optionLength == 4)
             {
                 await File.AppendAllTextAsync(arguments[3], report).ConfigureAwait(false);
             }
 
+            if (requireQuality && document.RootElement.GetProperty("status").GetString() !=
+                "measured_diagnostic_no_statistical_verdict")
+            {
+                Console.Error.WriteLine("Performance evidence is ineligible because quality did not match.");
+                return 3;
+            }
+
             return 0;
         }
         catch (Exception exception)

synapse.brainstorm.md

@@ -68,8 +68,9 @@ Date: 2026-09-27. Status: accepted for the bootstrap slice.
 
 - Keep functional `verify` fast and assertion-oriented. A long, exact Qwen
   continuation is not portable evidence of a quality regression when CPU
-  kernels choose a different valid sequence; test rejection with an impossible
-  reference token instead.
+  kernels choose a different valid sequence. Validate report and quality-gate
+  behavior from preserved raw evidence in TUnit; run the real matrix only in
+  the performance workflow.
 - Dispatch performance independently on macOS, Linux, and Windows. Preserve
   each runner's raw process samples and render a descriptive summary of the
   workload and medians. Hosted-runner results form separate hardware cohorts;

synapse.plan.md

@@ -129,11 +129,15 @@ gate and remote CI are not verified. No `TASK-BMK-001` release claim is closed.
 CI separation follow-up: Ubuntu `verify` on `3bbd5ee` ran 73 tests with one
 failure because a functional assertion required a platform-specific 32-token
 dotLLM divergence; the same test timed out after five minutes on hosted macOS.
-The quality-gate regression now uses an impossible expected token in an
-eight-token run, and the performance measurements are in a distinct
-manual Mac/Linux/Windows Actions workflow with raw artifacts and a C# report.
+The measured matrix is now removed from TUnit entirely: tests use recorded raw
+evidence for report and quality-gate behavior. Performance measurements are
+in a distinct manual Mac/Linux/Windows Actions workflow with raw artifacts
+and a C# report.
 The report behavior test was red (unknown `report` command) before the renderer
-was added. Three-OS CI execution, artifact publication, and the statistical
-release gate remain unverified until the new run completes.
+was added; the quality gate was separately red (usage exit 2) before its exit-3
+implementation. The first three-OS performance run (`9567d06`) passed and
+published three raw artifacts; Windows and Ubuntu verify passed while hosted
+Mac verify was still running at this checkpoint. The quality-gate follow-up
+and statistical release gate remain unverified until new runs complete.
 - [ ] Execute Qwen2 from the verified region IR instead of the parallel shadow
   loop, following `flybrain.plan.md` F0/F1 and accepted ADR-003.

tests/Synapse.IntegrationTests/Features/Benchmarking/DiagnosticMatrixReportTests.cs

@@ -35,4 +35,30 @@ public async Task RawMatrixEvidenceProducesDescriptivePerformanceReport()
         await Assert.That(report).Contains("peak RSS MiB");
         await Assert.That(report).Contains("Measured rounds: 5");
     }
+
+    [Test]
+    public async Task IneligibleRawEvidenceFailsPerformanceQualityGate()
+    {
+        var root = FindRepositoryRoot();
+        var evidence = Path.Combine(root, "benchmarks", "results",
+            "2026-09-28-m2-pro-qwen2.5-0.5b-q8_0-32tok-quality-divergence-final.json");
+        var start = new ProcessStartInfo("dotnet")
+        {
+            RedirectStandardOutput = true,
+            RedirectStandardError = true,
+            UseShellExecute = false,
+        };
+        start.ArgumentList.Add(Path.Combine(AppContext.BaseDirectory, "Synapse.ReferenceBenchmarks.dll"));
+        start.ArgumentList.Add("report");
+        start.ArgumentList.Add("--input");
+        start.ArgumentList.Add(evidence);
+        start.ArgumentList.Add("--require-quality");
+        using var process = Process.Start(start)
+            ?? throw new InvalidOperationException("Matrix report process did not start.");
+        var output = process.StandardOutput.ReadToEndAsync();
+        var error = process.StandardError.ReadToEndAsync();
+        await process.WaitForExitAsync().WaitAsync(TimeSpan.FromSeconds(30));
+        await Assert.That(process.ExitCode).IsEqualTo(3).Because(await error);
+        await Assert.That(await output).Contains("ineligible_quality_mismatch");
+    }
 }