Publish benchmark review and automate performance diagnostics

Author: KSemenenko

.github/workflows/performance.yml

@@ -1,6 +1,8 @@
 name: performance-diagnostic
 
 on:
+  push:
+    branches: [main]
   workflow_dispatch:
 
 permissions:

README.md

@@ -39,13 +39,13 @@ Details: [`docs/Architecture.md`](docs/Architecture.md).
 
 | Question | Answer |
 |---|---|
-| Does Synapse write as fast as **llama.cpp**? | **Yes.** 102% of its speed on 2 threads, 114% on 8 threads. |
-| Is a whole short request faster? | **Yes, 3.2× faster.** The model loads almost instantly. |
-| Does it use less memory? | **Yes, 2.3× less.** |
-| Does it start answering fast? | **Short prompts: yes. Long prompts: no, about 3× slower.** Main thing to fix. |
+| Does Synapse write as fast as **llama.cpp**? | **In the 8-token smoke, yes.** The native kernel reached 102% of llama.cpp at 2 threads and 114% at 8 threads. |
+| Is a whole short request faster? | **In that smoke, 3.2× faster.** Synapse's median fresh-process wall time was 216 ms versus 690 ms. |
+| Does it use less memory? | **In that smoke, 2.3× less peak RSS.** 555 MiB versus 1,256 MiB. |
+| Does it start answering fast? | **For the 5-token prompt, yes.** The older 274-token run was about 3× slower to first token; the new batched-prefill path still needs the same long test. |
 | Is it as fast as **MLX** on the Apple GPU? | **No.** MLX writes 225 tokens/s on the GPU. Synapse has no GPU code yet. |
-| How does **Microsoft Foundry Local** do? | **A separate test, 6 models.** With its own ONNX weights on about 6 cores, Qwen2.5 0.5B writes 231 tokens/s and the 7B models 22–25 tokens/s. |
-| Is it faster than the other .NET engine, dotLLM? | **Yes, 14× faster.** |
+| How does **Microsoft Foundry Local** do? | **Fast, but quality-limited in this test.** With separate ONNX weights on about 6 cores, Qwen2.5 0.5B reached 231 tokens/s and the 7B models 22–25 tokens/s; none of the 7 model/device variants fully completed the 128-token instruction. |
+| Is it faster than the other .NET engine, dotLLM? | **In the same 8-token smoke, 14× faster.** |
 
 ### Where it is tested
 
@@ -68,6 +68,20 @@ Details: [`docs/Architecture.md`](docs/Architecture.md).
 
 ![Synapse vs the reference engines on a MacBook Pro M2 Pro](docs/images/benchmark.svg)
 
+What the result says:
+
+- The Rust kernel produced the same eight tokens as llama.cpp in every
+  measured round and reached 136.6 versus 133.7 decode tokens/s at two
+  threads, then 190.6 versus 167.2 at eight threads.
+- The portable SIMD C# kernel reached 124.4 tokens/s at two threads, or 88%
+  of llama.cpp's 141.8. The Rust boundary is therefore useful in this measured
+  hotspot, while the C# path remains the tested portable implementation.
+- This is three warm-ups plus five measurements of an eight-token answer. It
+  proves short parity and supplies a diagnostic speed signal; it is not the
+  30-pair release verdict and does not establish long-answer quality.
+- The published long-prompt gap comes from an older build. Batched prefill is
+  implemented now, but its fresh GitHub long-answer result is still pending.
+
 ### GitHub Actions, committed code
 
 ![GitHub Actions performance run on macOS, Ubuntu, and Windows](docs/images/github-actions.svg)
@@ -88,14 +102,20 @@ Details: [`docs/Architecture.md`](docs/Architecture.md).
   up front (Phi-3.5-mini: 98 GiB, 12 s to the first token).
 - On GitHub, every runner and model pair is a separate job, and a model runs
   only where its file fits in half the RAM.
+- Manual review of the preserved 128-token outputs found no fully compliant
+  result among the seven model/device variants. Qwen3 and DeepSeek-R1 spent the
+  budget on reasoning and emitted no final answer; Qwen2.5 CPU and Phi-3.5
+  contained factual errors; Phi-4, Mistral, and Qwen2.5 WebGPU left required
+  sections or the three-line recap incomplete. These bars measure throughput,
+  not answer quality, and are never ranked against the GGUF or MLX cohorts.
 
 ### What to improve, and where
 
 | | Problem | Fix | Where |
 |---|---|---|---|
-| 🔴 | Long prompts: first token about 3× slower | Read the prompt in batches (task CPU.2) | `Qwen2CpuExecutor.Prefill` |
+| 🔴 | Older long-prompt run: first token about 3× slower | Rerun the same matrix on the new batched-prefill path | `performance.yml` |
 | 🔴 | No GPU | Metal kernels | not started |
-| 🟡 | GitHub still measures the old scalar build | Commit the new kernels, rerun the performance workflow | `.github/workflows/performance.yml` |
+| 🟡 | Hosted evidence for the new kernels is pending | Complete and review the current performance workflow | `.github/workflows/performance.yml` |
 | 🟡 | Takes token IDs, not text | Tokenizer and chat templates | not started |
 | 🟡 | Only Qwen2 runs | More model families | `src/Synapse.Runtime/Features/TextGeneration` |
 | 🟢 | Speed, memory, startup | Confirm with the 30-run release gate | `experiments/Synapse.ReferenceBenchmarks` |

benchmarks/README.md

@@ -63,8 +63,10 @@ runs A–K.
 Each model and scenario ran in a fresh process. The model stayed loaded for
 1 warm-up and 3 measured requests; every request was a new chat session with
 the full transcript. Context was capped at 1,024 tokens (see below). Greedy
-decoding, answer quality not reviewed. AC power; other desktop apps were open
-(load average about 3.4).
+decoding. AC power; other desktop apps were open (load average about 3.4).
+The raw schema still says `quality_unreviewed`; the manual review below was
+performed afterward from the preserved text and reasoning fields and does not
+retroactively change measurement eligibility.
 
 128-token answer (`capitals-single-long`), median of 3:
 
@@ -100,6 +102,29 @@ turn 3, median of 3:
 - Prompt token counts differ because each model has its own tokenizer and
   chat template.
 
+### Manual output review
+
+No model/device variant fully satisfied the 128-token instruction. This makes
+the throughput table useful for runtime diagnosis, not for a quality-adjusted
+model ranking.
+
+| Variant | Review of the deterministic measured output |
+|---|---|
+| Qwen2.5 0.5B CPU | Incorrect and incomplete: it names all three capitals, but calls Fort Knox the federal headquarters and misidentifies Paris landmarks; it hits the 128-token cap before the recap. |
+| Qwen2.5 0.5B WebGPU | Mostly factual but incomplete: it stops at 99 tokens without the requested three-line recap. |
+| Qwen3 0.6B CPU | Incorrect and incomplete: all 128 tokens are reasoning, it emits no final answer, and the reasoning calls Paris, London, and Madrid France's capitals. |
+| Phi-3.5-mini CPU | Incorrect and incomplete: it corrupts Washington, D.C., places Mount Rushmore near it, and reaches the cap during the recap. |
+| Phi-4-mini CPU | The three main country-capital sections are substantially correct, but the generated recap is malformed and truncated at the cap. |
+| Mistral 7B v0.2 CPU | The France section is correct, but the output reaches the cap during the United States section before covering the United Kingdom or recap. |
+| DeepSeek-R1 7B CPU | Incomplete: all 128 tokens are reasoning and no final answer is emitted, although the reasoning recalls the three capitals. |
+
+The three-turn outputs have the same constraint: every CPU turn reaches its
+64-token cap. DeepSeek-R1 and Qwen3 remain reasoning-only. Phi-3.5, Phi-4, and
+Mistral contain mostly correct briefing content but truncate sections; the
+Qwen2.5 variants additionally contain errors such as placing Paris in the
+center or south of France and describing Washington transit as free. No
+instruction-compliance claim is made from those dialogue timings.
+
 **Context cap.** Every Foundry package sets its context length to the model's
 full window, and ONNX Runtime GenAI reserves the KV memory for that whole
 window at the first request. With the package default, Qwen3 0.6B reached

docs/ADR/ADR-005-benchmark-timing-provenance.md

@@ -37,13 +37,13 @@ invalid. A missing platform metric remains null with its scope recorded.
 
 Functional CI and performance evidence are separate workflows. Functional
 TUnit checks can validate rendering against recorded raw evidence, but do not
-run the measured matrix or assert throughput. The manually
-dispatched performance workflow runs its own macOS/Linux/Windows matrix and
-retains every measured round as an artifact with a descriptive report. An
-output mismatch fails the job after writing the report, while the raw artifact
-is still uploaded. Hosted
-runners are variable hardware, so their results are never pooled across OSes
-or compared to the owner's Mac as a release winner verdict.
+run the measured matrix or assert throughput. The performance workflow runs
+automatically for every push to `main` and also supports manual dispatch. It
+runs its own macOS/Linux/Windows matrix and retains every measured round as an
+artifact with a descriptive report. An output mismatch fails the job after
+writing the report, while the raw artifact is still uploaded. Hosted runners
+are variable hardware, so their results are never pooled across OSes or
+compared to the owner's Mac as a release winner verdict.
 
 ## Consequences
 

docs/Features/Benchmarking.md

@@ -61,8 +61,9 @@ the functional suite on macOS ARM64, Ubuntu x64, and Windows x64; it does not
 run the measured matrix or enforce speed thresholds. Existing recorded JSON
 tests report formatting and mismatch rejection without running performance
 samples inside TUnit.
-`performance.yml` is a separate manually dispatched three-OS CPU matrix. It
-fetches only the pinned Qwen smoke GGUF, builds the pinned competitors, runs
+`performance.yml` is a separate three-OS CPU matrix that runs on every push
+to `main` and supports manual dispatch. It fetches only the pinned Qwen smoke
+GGUF, builds the pinned competitors, runs
 three warm-ups plus five measured fresh-process rounds with a two-thread cap,
 and uploads unmodified per-round JSON for each runner. The C# `report --input
 <raw.json> [--summary <path>] [--require-quality]` command renders workload, model digest,
@@ -197,8 +198,13 @@ cohorts. Weights are unchanged.
 
 Limits: the SDK does not expose ONNX Runtime thread settings, so this cohort
 is not thread-capped. Its weights and quantization differ from GGUF Q8_0 and
-MLX 8-bit, so it is never ranked against those cohorts. Answer quality is
-unreviewed, and reasoning text is stored apart from answer text.
+MLX 8-bit, so it is never ranked against those cohorts. Raw evidence retains
+the original `quality_unreviewed` status and stores reasoning apart from
+answer text. A later manual documentation review found that none of the seven
+local model/device variants fully completed the 128-token instruction: two
+were reasoning-only, two contained material factual errors, and the remaining
+three omitted or truncated required sections. That review is descriptive and
+does not make the run eligible for a quality-adjusted verdict.
 
 ### Requirements
 

synapse.brainstorm.md

@@ -138,3 +138,16 @@ Date: 2026-09-27. Status: accepted for the bootstrap slice.
   model, explicit download, raw JSON per job.
 - Rejected: reusing one session per scenario (hidden KV reuse), the OpenAI
   client (no TTFT), and adding the SDK to the existing runner (RID leak).
+
+## 2026-09-28 public benchmark interpretation
+
+- Scope every public CPU claim to the preserved eight-token diagnostic: the
+  native backend reached 102% of llama.cpp at two threads and 114% at eight,
+  with exact eight-token parity. This is not a long-answer or 30-pair verdict.
+- Treat the existing 274-token first-token gap as evidence from the older
+  pre-batched-prefill build. The implementation changed; only a fresh run can
+  confirm whether the bottleneck moved.
+- Review Foundry output separately from throughput. The manual review found
+  reasoning-only, factually wrong, and truncated answers; none of the seven
+  model/device variants completed the 128-token instruction. Keep the raw
+  evidence status unchanged and publish the review as a descriptive layer.

synapse.plan.md

@@ -133,6 +133,16 @@ files plus two package-default probes). The full .NET suite passed 189/189
 with the local dotLLM and Homebrew llama.cpp `b29c606e2` subjects. The 15
 hosted Foundry jobs and the `verify.yml` anchor fetch have not run.
 
+Public-results checkpoint (2026-09-28): README and benchmark documentation
+claims are now scoped to the raw evidence. The native CPU result is presented as an
+eight-token diagnostic (102% of llama.cpp at two threads, 114% at eight), the
+older long-prompt gap is not attributed to the new batched-prefill build, and
+the manually reviewed Foundry outputs are reported as quality-ineligible: no
+model/device variant fully completed the 128-token instruction. Performance
+diagnostics now trigger on every push to `main` as well as manual dispatch.
+This does not close `TASK-BMK-001`, `TASK-BMK-002`, or any CPU task; fresh
+hosted evidence is still required.
+
 Local direct-native evidence: pinned Homebrew llama.cpp `b29c606e2` reproduced
 the Qwen prompt token IDs and continuation, three focused real-process native
 regressions passed, and raw 8/128-token samples plus a separate `llama-bench`

tests/Synapse.IntegrationTests/Features/Benchmarking/FoundryLocalPlanTests.cs

@@ -77,6 +77,8 @@ public async Task WorkflowsIsolateFoundryLocalJobs()
         var performance = await File.ReadAllTextAsync(Path.Combine(workflows, "performance.yml"));
         var verify = await File.ReadAllTextAsync(Path.Combine(workflows, "verify.yml"));
 
+        await Assert.That(performance).Contains("push:\n    branches: [main]");
+        await Assert.That(performance).Contains("workflow_dispatch:");
         await Assert.That(performance).Contains("foundry-local-plan:");
         await Assert.That(performance).Contains("matrix: ${{ fromJSON(needs.foundry-local-plan.outputs.matrix) }}");
         await Assert.That(performance).Contains("runs-on: ${{ matrix.runner }}");