You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 181d22b
Browse filesBrowse the repository at this point in the historyBrowse files
| Is a whole short request faster? |**Yes in the 8-token smoke.** The local median was 216 ms versus 690 ms; hosted medians were 526–557 ms versus 635–1,671 ms. |
44
44
| Does it use less memory? |**Yes in these CPU diagnostics.** The local smoke was 555 MiB versus 1,256 MiB; hosted Synapse used 539–553 MiB versus 575–1,203 MiB. |
45
45
| Does it start answering fast? |**For the 5-token smoke prompt, yes.** On the hosted long prompt, Synapse's first token was slower than LLamaSharp on all three OSes even with batched prefill. |
46
-
| Is it as fast as **MLX** on the Apple GPU? |**No.** MLX writes 225 tokens/s on the GPU. The new Synapse Metal backend wrote 135 tokens/s in one local 128-token run (llama.cpp Metal: 157). |
46
+
| Does Synapse compute the same thing as **llama.cpp**? |**Yes.** On identical tokens, perplexity matches within 0.01% from 4k to 32k of context, and on the Apple GPU Synapse writes the same tokens (128 of 128 at 4k–16k). |
47
+
| Is it as fast on the **Apple GPU** with long context? |**Writing, yes; the first token, not yet.** From 4k to 32k Synapse writes 90–140 tokens/s, level with llama.cpp (93–141); MLX falls to 60 at 32k. The first token takes 1.2–1.6× llama.cpp's time and 1.6–2.1× MLX's. |
48
+
| Does it use less memory with long context? |**Same as llama.cpp, 25× less than MLX.** At 32k: 533 MiB peak footprint against 535 (llama.cpp) and 13,500 (MLX). |
49
+
| Is the model still right on long prompts? |**As right as a 0.5B model gets.** Synapse, llama.cpp, and MLX give the same answers; the misses (a hidden number at 32k, keys among distractors, variable chains) happen in every engine, so they are the model's limits. |
47
50
| How does **Microsoft Foundry Local** do? |**Fast, but quality-limited in this test.** With separate ONNX weights on about 6 cores, Qwen2.5 0.5B reached 231 tokens/s and the 7B models 22–25 tokens/s; none of the 7 model/device variants fully completed the 128-token instruction. |
48
51
| Is it faster than the other .NET engine, dotLLM? |**In the same 8-token smoke, 14× faster.**|
49
52
@@ -100,12 +103,22 @@ these diagnostics do not pass the 30-pair release gate. The performance
100
103
workflow now assembles all raw artifacts into one results-table artifact and
101
104
its final GitHub job summary.
102
105
103
-
### Long prompts
106
+
### Long context on the Apple GPU
104
107
105
-
The plot below is the earlier scalar diagnostic. Current hosted results are
106
-
in the table above and the benchmark details.
108
+
107
109
108
-
110
+
-**Tokens.** Every engine gets the same prompt, and its token IDs are
111
+
checked before its answer counts. Synapse's answers match llama.cpp's
112
+
byte for byte in 28 of 28 long-context questions.
113
+
-**Memory.** Synapse keeps the KV cache at 384 MiB for 32k tokens (FP16) and
114
+
maps the model file instead of copying it.
115
+
-**Speed.** Writing is level with llama.cpp. The first token is the open gap:
116
+
prompt attention and the prompt matrix multiply are slower. Removing the
117
+
staged K/V tile from prompt attention already cut 32k from 40.5 s to 32.6 s.
118
+
-**FP32 KV cache.** Synapse writes 73 tokens/s at 32k, against 10.7 for
119
+
llama.cpp with the same cache type.
120
+
- Details, every engine and KV cache type, and CPU rows: runs Q–T in
121
+
[`benchmarks/README.md`](benchmarks/README.md).
109
122
110
123
### Microsoft Foundry Local, 6 models
111
124
@@ -131,9 +144,11 @@ in the table above and the benchmark details.
131
144
|| Problem | Fix | Where |
132
145
|---|---|---|---|
133
146
| 🔴 | Hosted long-prompt first token and 128-token decode trail the CPU references | Profile prefill and decode on each runner |`experiments/Synapse.ReferenceBenchmarks`|
134
-
| 🟡 | Metal runs Qwen2 but decodes slower than llama.cpp and MLX; CUDA is written but has never run | FP16 KV cache, GPU-packed weights; run CUDA on an NVIDIA machine |`native/synapse-gpu`|
135
-
| 🟡 | Long-output quality is unreviewed | Review generated text and add a quality gate before release claims |`.github/workflows/performance.yml`|
136
-
| 🟡 | Takes token IDs, not text | Tokenizer and chat templates | not started |
147
+
| 🔴 | Apple GPU first token is 1.2–1.6× llama.cpp's; the prompt matrix multiply alone is 1.55× slower at 512 tokens | Profile the GEMM with GPU counters (tile, half, and layout experiments did not close it) |`native/synapse-gpu/.../metal/shaders/matmul.metal`|
148
+
| 🟡 | CPU first token is 1.9× llama.cpp's at 8k | Tiled, vectorized prompt attention |`Qwen2CpuAttention.cs`|
149
+
| 🟡 | Smart context (query-aware KV pages) is proven only on CPU: +3.4% perplexity while reading 30% of an 8k context | Metal kernels matched to the C# selection, then measure speed and memory | ADR-016, `KvPageSelector.cs`|
150
+
| 🟡 | CUDA is written but has never run | Run CUDA on an NVIDIA machine |`native/synapse-gpu`|
151
+
| 🟡 | Free-form long answers are checked for agreement with llama.cpp, not reviewed for content | Human review plus exact-answer gates before release claims |`.github/workflows/performance.yml`|
137
152
| 🟡 | Only Qwen2 runs | More model families |`src/Synapse.Runtime/Features/TextGeneration`|
138
153
| 🟢 | Speed, memory, startup | Confirm with the 30-run release gate |`experiments/Synapse.ReferenceBenchmarks`|
| M. Hosted CPU, MLX, and Foundry, 3 operating systems | Rust CPU kernels for Synapse | CPU 2 / external runtime default | CPU smoke 3+5; long/MLX/Foundry 1+3 |[GitHub run `36435838291`](https://github.com/managedcode/Synapse/actions/runs/36435838291), 22 raw artifacts |
26
26
| N. Metal GPU long context, 40k and 131k synthetic tokens | Metal backend (ADR-012), FP32 KV | GPU | 1 measured per row |[JSON](results/2026-09-28-m2-pro-qwen2.5-0.5b-q8_0-metal-long-context-speed-diagnostic.json)|
27
-
| O. Metal GPU pass-key retrieval, 4k to 120k | Metal backend, FP32 KV (native window) and FP16 KV (YaRN ×4) | GPU | 1 measured per row |[native](results/2026-09-28-m2-pro-qwen2.5-0.5b-q8_0-metal-passkey-native-f32-diagnostic.json), [YaRN](results/2026-09-28-m2-pro-qwen2.5-0.5b-q8_0-metal-passkey-yarn4-f16-diagnostic.json)|
27
+
| O. Metal GPU pass-key retrieval, 4k to 120k | Metal backend, FP32 KV (native window) and FP16 KV (YaRN ×4) | GPU | 1 measured per row |[native](results/2026-09-28-m2-pro-qwen2.5-0.5b-q8_0-metal-passkey-native-f32-diagnostic.json), [YaRN](results/2026-09-28-m2-pro-qwen2.5-0.5b-q8_0-metal-passkey-yarn4-f16-diagnostic.json), [llama.cpp control, corrected](results/2026-09-28-m2-pro-llamacpp-metal-passkey-120k-control-diagnostic.json)|
28
+
| P. Tokenizer parity, whole repository | Repo-owned BPE tokenizer (ADR-014) | — | 702 runs, both special modes |[JSON](results/2026-09-28-m2-pro-qwen2.5-tokenizer-corpus-parity.json)|
29
+
| Q. Perplexity parity with llama.cpp, 4k to 32k | Metal FP32/FP16 KV, native CPU | GPU / CPU 8 | 1 measured per row |[JSON](results/2026-09-28-m2-pro-qwen2.5-0.5b-q8_0-perplexity-parity-diagnostic.json)|
30
+
| R. Long-context answers, 4 engines, 4k to 32k | Metal FP32/FP16 KV, native CPU | GPU / CPU 8 | 1 measured per case |[GPU](results/2026-09-28-m2-pro-qwen2.5-0.5b-q8_0-long-context-quality-diagnostic.json), [CPU](results/2026-09-28-m2-pro-qwen2.5-0.5b-q8_0-long-context-quality-cpu-diagnostic.json)|
31
+
| S. KV page activation quality (ADR-016) | Native CPU, C# page selection | CPU 8 | 1 measured per row |[JSON](results/2026-09-28-m2-pro-qwen2.5-0.5b-q8_0-kv-page-activation-cpu-diagnostic.json)|
32
+
|**T. Context sweep, 4k to 32k, tokens + memory + speed (main README)**| Metal FP32/FP16 KV with prompt attention reading K/V directly; native CPU | GPU / CPU 8 | 1 warm-up + 2 measured, rotated |[JSON](results/2026-09-28-m2-pro-qwen2.5-0.5b-q8_0-context-sweep-diagnostic.json), [prompt IDs](scenarios/context-sweep/)|
28
33
29
34
Local machine: MacBook Pro, Apple M2 Pro (8 performance + 4 efficiency cores,
30
35
19-core GPU), 32 GB, macOS 27.0 arm64. Every CPU sample starts a new process.
"output": " The grass is green. The sky is blue. The sun",
13
+
"output": "6",
14
14
"containsAnswer": false,
15
-
"promptEvalMilliseconds": 288020.20,
16
-
"promptTokensPerSecond": 416.60,
17
-
"decodeMillisecondsPerToken": 20.86,
18
-
"decodeTokensPerSecond": 47.94,
19
-
"timeLPeakMemoryFootprintBytes": 1954515680,
15
+
"promptEvalMilliseconds": 458669.90,
16
+
"promptTokensPerSecond": 261.61,
17
+
"decodeMillisecondsPerToken": null,
18
+
"decodeTokensPerSecond": null,
19
+
"timeLPeakMemoryFootprintBytes": 1981958880,
20
20
"kvCache": "llama.cpp default (F16)",
21
-
"note": "Same token IDs as the Synapse run that answered only 6; both engines miss the 10%-depth key at 120k, which points to the model under YaRN x4, not to either engine."
21
+
"note": "On truly identical token IDs llama.cpp answers exactly like Synapse: the single digit 6, then end of turn, instead of 68317. Both engines agree; the miss is the model under YaRN x4.",
22
+
"promptTokensEvaluatedByLlamaCpp": 119991,
23
+
"timingNote": "Recorded while a CPU experiment ran in parallel; not speed evidence. The first run (288,020 ms, 416.6 tok/s) had one token fewer.",
24
+
"correction": {
25
+
"date": "2026-09-28",
26
+
"finding": "The first run of this control passed prompt.txt with -f. llama.cpp -f drops one trailing newline, so it evaluated 119,990 tokens (its own log: prompt eval time = 288020.20 ms / 119990 tokens) although this file recorded 119,991 identical IDs. Without the newline after <|im_start|>assistant the model printed filler text.",
27
+
"fix": "The rerun passes the prompt text plus one extra newline; llama.cpp then reports 119,991 prompt tokens, and llama-tokenize --no-escape of the prompt text equals the Synapse IDs.",
28
+
"firstRunOutput": " The grass is green. The sky is blue. The sun"
0 commit comments