Add layer-drop, speculation, and K-quant work

Author: KSemenenko

README.md

@@ -26,8 +26,10 @@ weight sits in memory. Synapse runs a model like a fly brain.
 - **Regions on different machines.** Only the hidden state travels between
   them: 3.5 KiB per token for Qwen2.5 0.5B.
 
-Today a dense Qwen2.5 runs end to end. Skipping, on-demand loading, and the
-cluster are next.
+Today Qwen2.5 0.5B and 7B run end to end: Q8_0 on the CPU and the Apple GPU,
+Q4_K_M on the Apple GPU. A dropped layer is never loaded, with its quality cost measured
+first, and a small model drafts for a big one without changing its answer.
+Router-driven skipping, expert paging, and the cluster are next.
 
 ## Architecture
 
@@ -47,6 +49,10 @@ Details: [`docs/Architecture.md`](docs/Architecture.md).
 | Is it as fast on the **Apple GPU** with long context? | **Writing, yes; the first token, nearly.** From 4k to 32k Synapse writes 92–148 tokens/s, level with llama.cpp (93–141); MLX falls to 60 at 32k. The first token takes 1.04× llama.cpp's time at 32k and 1.5× at 4k; MLX is 1.6–1.8× faster to the first token. |
 | Does it use less memory with long context? | **Less than llama.cpp when the window is large, 25× less than MLX.** At 32k: 534 MiB peak against 535 (llama.cpp) and 13,500 (MLX). With a 131k window and a 4k prompt: 234 MiB against 1,645, because Synapse's KV cache grows with use. |
 | Is a second question about the same long document fast? | **Yes, with prefix reuse.** A second question about a 30,000-token document starts answering in 0.13 s instead of 23.8 s, with the same answer. |
+| Does it run a 7B model on the Apple GPU? | **Yes.** Qwen2.5-7B-Instruct-1M Q8_0 writes llama.cpp's exact text at the same decode speed (17.5 tokens/s). The first token of a 3,528-token prompt is 1.5× slower (17–19 s against 11.5 s). |
+| Does a Q4_K_M model run, and how small is it? | **Yes.** Qwen2.5-7B-Instruct-1M Q4_K_M peaks at 4.2 GB in Synapse, against 4.8 GB in llama.cpp and 7.2 GB for Q8_0, and writes llama.cpp's text. Its decode is 0.64× llama.cpp's (18 against 28 tokens/s). |
+| Does the model have to be loaded whole? | **No.** A dropped layer's weights and KV are never loaded; each of the 7B's layers is about 240 MiB. Only 2 of its 28 layers are free to drop (−1.1% perplexity), and 8 cost +137%. Quantization saves more: the same model in Q4_K_M peaks at 4.2 GB. |
+| Can a small model speed up a big one? | **Yes, with the same output.** Qwen2.5-0.5B drafts and the 7B checks: +23–35% decode on chat text. On code-heavy text the draft agrees too rarely to help. |
 | Is the model still right on long prompts? | **As right as a 0.5B model gets.** Synapse, llama.cpp, and MLX give the same answers; the misses (a hidden number at 32k, keys among distractors, variable chains) happen in every engine, so they are the model's limits. |
 | How does **Microsoft Foundry Local** do? | **Fast, but quality-limited in this test.** With separate ONNX weights on about 6 cores, Qwen2.5 0.5B reached 231 tokens/s and the 7B models 22–25 tokens/s; none of the 7 model/device variants fully completed the 128-token instruction. |
 | Is it faster than the other .NET engine, dotLLM? | **In the same 8-token smoke, 14× faster.** |
@@ -129,6 +135,28 @@ its final GitHub job summary.
 - Details, every engine and KV cache type, and CPU rows: runs Q–V in
   [`benchmarks/README.md`](benchmarks/README.md).
 
+### A 7B model, not loaded whole
+
+![Qwen2.5-7B on the Apple GPU: memory, speed, and first token](docs/images/large-model.svg)
+
+- **Runs.** Qwen2.5-7B-Instruct-1M (head size 128) runs on Metal from Q8_0 and Q4_K_M files. For the pinned
+  prompt it writes llama.cpp's text with both.
+- **Q4_K_M** (ADR-021). The same model in Q4_K_M peaks at 4.2 GB instead of 7.3, with no perplexity loss on
+  the test text. It uses less memory than llama.cpp, but decodes at 0.64× llama.cpp's speed. That kernel is the
+  next target.
+- **Drop layers, never load them** (ADR-019). A qualified drop profile skips whole layers; their weights are
+  never mapped for the GPU and their KV is never allocated.
+
+  | Layers dropped (of 28) | Peak RSS | Perplexity | Same top-1 as dense |
+  |---:|---:|---:|---:|
+  | 0 | 7,255 MiB | 12.16 | 100% |
+  | 2 (qualified) | 6,780 MiB | −1.1% | 80% |
+  | 4 | 6,326 MiB | +39% | 66% |
+  | 8 | 5,381 MiB | +137% | 51% |
+- **Small drafts, big checks** (ADR-020). The 0.5B proposes 3 tokens and the 7B verifies them in one pass. On
+  chat text the 7B Q8_0 decodes at 23 instead of 18 tokens/s, and the output is exactly the 7B's.
+- Details: runs W, X, and Y in [`benchmarks/README.md`](benchmarks/README.md).
+
 ### Microsoft Foundry Local, 6 models
 
 ![Microsoft Foundry Local: six models from four families on the Mac](docs/images/foundry-local.svg)
@@ -154,7 +182,9 @@ its final GitHub job summary.
 |---|---|---|---|
 | 🔴 | Hosted long-prompt first token and 128-token decode trail the CPU references | Profile prefill and decode on each runner | `experiments/Synapse.ReferenceBenchmarks` |
 | 🔴 | Apple GPU first token is 1.5× llama.cpp's at 4k (1.04× at 32k); the prompt matrix multiply alone is 1.55× slower at 512 tokens | Profile the GEMM with GPU counters (tile, half, and layout experiments did not close it) | `native/synapse-gpu/.../metal/shaders/matmul.metal` |
-| 🟡 | Past 32k the 0.5B model is out of its rated range | Head size 128 kernels, then Qwen2.5-7B-Instruct-1M | `native/synapse-gpu`, `gpu-kernels.plan.md` LONG.1 |
+| 🔴 | Q4_K_M decode is 0.64× llama.cpp's and prefill 1.6× slower (the memory is already lower) | A pipelined K-quant GEMM and a `mul_mv_q4_K`-style matrix-vector kernel | `gpu-kernels.plan.md` Q4K.2 |
+| 🟡 | A speculative check of 6–8 tokens costs about three single-token passes | A small-batch kernel that reads each weight once | `metal/shaders/matmul.metal` |
+| 🟡 | The 7B's first token is 1.5× llama.cpp's on a 3.5k prompt | The same GEMM work as the 0.5B | `gpu-kernels.plan.md` LONG.1 |
 | 🟡 | Prefix reuse lives in one process | Persistent prefix cache: KV on disk, metadata in ZoneTree | ADR-018 |
 | 🟡 | CPU first token is 1.9× llama.cpp's at 8k | Tiled, vectorized prompt attention | `Qwen2CpuAttention.cs` |
 | 🟡 | Smart context (query-aware KV pages) is proven only on CPU: +3.4% perplexity while reading 30% of an 8k context | Metal kernels matched to the C# selection, then measure speed and memory | ADR-016, `KvPageSelector.cs` |
@@ -170,6 +200,7 @@ All runs and raw data: [`benchmarks/README.md`](benchmarks/README.md).
 ![Synapse progress and roadmap](docs/images/roadmap.svg)
 
 Plan files: [`synapse.plan.md`](synapse.plan.md),
+[`gpu-kernels.plan.md`](gpu-kernels.plan.md),
 [`cpu-kernels.plan.md`](cpu-kernels.plan.md),
 [`flybrain.plan.md`](flybrain.plan.md),
 [`elastic-inference.plan.md`](elastic-inference.plan.md),

benchmarks/README.md

@@ -32,6 +32,9 @@ measures Microsoft Foundry Local alone on six models from four families.
 | **T. Context sweep, 4k to 32k, tokens + memory + speed (main README)** | Metal FP32/FP16 KV with prompt attention reading K/V directly; native CPU | GPU / CPU 8 | 1 warm-up + 2 measured, rotated | [JSON](results/2026-09-28-m2-pro-qwen2.5-0.5b-q8_0-context-sweep-diagnostic.json), [prompt IDs](scenarios/context-sweep/) |
 | U. Dynamic KV memory, 131k window (ADR-017) | Metal FP16/FP32 KV, growing slots with reservation | GPU | 1 measured per row | [JSON](results/2026-09-29-m2-pro-qwen2.5-0.5b-q8_0-dynamic-kv-memory-diagnostic.json) |
 | V. Second question about a 30k-token document (ADR-018) | Metal FP16 KV, prompt prefix reuse | GPU | 1 measured per row | [JSON](results/2026-09-29-m2-pro-qwen2.5-0.5b-q8_0-prefix-reuse-diagnostic.json) |
+| W. Qwen2.5-7B-1M: layer drop quality and memory (ADR-019) | Metal FP16 KV, head size 128, weight segments | GPU | 1 run per drop set; memory 1–4 runs | [qualification 7B](results/2026-09-29-m2-pro-qwen2.5-7b-1m-q8_0-layer-drop-qualification.json), [qualification 0.5B](results/2026-09-29-m2-pro-qwen2.5-0.5b-q8_0-layer-drop-qualification.json), [memory](results/2026-09-29-m2-pro-qwen2.5-7b-1m-q8_0-layer-drop-memory-diagnostic.json) |
+| Y. Qwen2.5-7B-1M Q4_K_M against Q8_0 and llama.cpp (ADR-021) | Metal FP16 KV, K-quant kernels | GPU | 2 alternating rounds | [JSON](results/2026-09-29-m2-pro-qwen2.5-7b-1m-k-quant-diagnostic.json) |
+| X. Qwen2.5-7B-1M with a 0.5B draft (ADR-020) | Metal FP16 KV, exact greedy speculation | GPU | 2 alternating rounds | [JSON](results/2026-09-29-m2-pro-qwen2.5-7b-1m-speculative-decoding-diagnostic.json) |
 
 Local machine: MacBook Pro, Apple M2 Pro (8 performance + 4 efficiency cores,
 19-core GPU), 32 GB, macOS 27.0 arm64. Every CPU sample starts a new process.
@@ -311,6 +314,74 @@ haystack. One model asks two questions, 32 new tokens each.
   persistent prefix cache across processes is the next step.
 - One run per row: a diagnostic, not a paired verdict.
 
+## W. Qwen2.5-7B-Instruct-1M: dropping layers (ADR-019)
+
+**Quality.** Perplexity over 1,024 scored positions of the pinned haystack (`experiments layer-drop`); dense
+12.163. Top-1 is the share of positions whose arg-max equals the dense model's.
+
+| Dropped layers | Perplexity change | Top-1 as dense |
+|---|---:|---:|
+| 12 | −3.6% | 86.5% |
+| 11 | −2.5% | 85.7% |
+| 8 | +0.8% | 84.6% |
+| 0 (the first) | +2,467,089% | 0.3% |
+| 27 (the last) | +251% | 67.5% |
+| 11, 12 (qualified) | −1.1% | 79.8% |
+| 8, 11, 12, 14 | +39.2% | 66.4% |
+| 8 middle layers | +137.5% | 50.9% |
+
+- A single dropped layer that lowers perplexity still changes about 14% of the predictions.
+- The 0.5B has no cheap layer: the best single drop costs +7.6%.
+
+**Memory and speed.** 3,528-token prompt, 128 tokens, one fresh process per row.
+
+| Variant | Peak RSS | First token | Decode tok/s |
+|---|---:|---:|---:|
+| Synapse dense | 7,255 MiB | 17.6 s | 13.3–17.0 |
+| Synapse, 2 layers dropped (qualified) | 6,780 MiB | 17.5 s | 13.9–15.0 |
+| Synapse, 4 layers dropped | 6,326 MiB | 16.0 s | 14.7 |
+| Synapse, 8 layers dropped | 5,381 MiB | 13.3 s | 17.9 |
+| llama.cpp Q8_0 | 8,096 MiB | 11.5 s | 18.6 |
+| llama.cpp Q4_K_M | 4,742 MiB | 12.2 s | 29.2 |
+
+- Metal wraps only the weight segments the kept layers use; before that, a 2-layer drop still peaked at
+  7,271 MiB.
+- Decode varied by ±12% across identical runs on a warm machine, so only the memory column compares run
+  by run.
+
+## X. Qwen2.5-7B-Instruct-1M with a Qwen2.5-0.5B draft (ADR-020)
+
+A 37-token chat prompt and 256 tokens; rounds alternate dense and speculative runs.
+
+| Draft tokens | Decode tok/s, round 1 / 2 | Acceptance | Tokens per 7B pass |
+|---:|---:|---:|---:|
+| — (dense) | 17.4 / 18.3 | — | 1 |
+| 3 | 23.5 / 22.5 | 54% | 2.63 |
+| 5 | 14.0 / 13.5 | 40% | 3.00 |
+| 7 | 14.2 / 13.9 | 32% | 3.27 |
+
+- The output is the 7B's own greedy output. The token identity of the two models is checked before
+  running.
+- A 4-token check costs about 1.7 single-token passes; 6–8 tokens cost about 3, so 3 draft tokens is the
+  default.
+- On the repository-summary prompt acceptance was 27–46% and speculation was slower than dense.
+
+## Y. Qwen2.5-7B-Instruct-1M Q4_K_M (ADR-021)
+
+The same file runs in both engines. 3,528-token prompt, 128 tokens, two alternating rounds.
+
+| Engine and weights | Peak RSS | First token | Decode tok/s |
+|---|---:|---:|---:|
+| Synapse Q4_K_M | 4,233–4,248 MiB | 19.9–20.1 s | 17.4–18.2 |
+| llama.cpp Q4_K_M | 4,846–4,851 MiB | 12.1–12.2 s | 27.0–29.3 |
+| Synapse Q8_0 | 7,210–7,252 MiB | 16.9–17.3 s | 15.8–17.9 |
+| llama.cpp Q8_0 | 7,866–8,109 MiB | 10.8–11.2 s | 17.3–17.7 |
+
+- For the pinned prompt, Synapse Q4_K_M writes the same text as llama.cpp.
+- Perplexity on the same 2,048 tokens is 11.93 for Q4_K_M against 12.16 for Q8_0.
+- With the 0.5B draft, Q4_K_M decode drops to 14.4–14.8 tokens/s: acceptance is 40%, and a 4-token check
+  still costs too much.
+
 ## P–S. Quality on long prompts (ADR-014, ADR-015, ADR-016)
 
 Details and tables: [`docs/Features/LongContext.md`](../docs/Features/LongContext.md)