From 53e6697ad6b4d67e9ef9edd8d4e99220e3dcc865 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 7 Jul 2026 03:01:01 +0000 Subject: [PATCH] docs(readme): fix stale TTFT claim, clarify reuse-ceiling framing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two stale claims carried over from the same disproven benchmark the blog post and paper.md already corrected: - The "Cold short-prompt TTFT" row claimed Ollama beat Squish 167ms vs 192ms. That number came from a flawed same-fixed-prompt-repeated-5x methodology that was actually measuring Ollama's own cache hit, not a genuinely cold comparison. Replaced with the corrected cold/unique floor benchmark: Squish leads 800ms vs 812ms at 75 tokens. - The "What Squish Doesn't Do" section repeated the same disproven 167/192 claim as a limitation. Removed — Squish has no measured TTFT loss under real cold, unique-prompt conditions. - Added a footnote clarifying the 9.8x figure is the exact-repeat reuse ceiling, not the typical case, and pointed to the actual 1.15-1.32x floor for unique prompts. --- README.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 327524ad..58ec3914 100644 --- a/README.md +++ b/README.md @@ -64,15 +64,17 @@ Measured on an Apple **M3 MacBook Pro, 16 GB** — **thermally controlled** (eac | Metric | Ollama | **Squish** | |---|---:|---:| | **Cold start** — load + first token (1.5B) | 20–30 s | **≈ 0.5 s**  _(54× load)_ | -| **Full response** @ 4000-token prompt | 37.5 s | **3.8 s**  _(9.8× faster)_ | +| **Full response** @ 4000-token prompt (repeated exactly)\* | 37.5 s | **3.8 s**  _(9.8× faster)_ | | **Decode throughput** @ 75 tokens | 20.3 tok/s | **24.0 tok/s**  _(INT3)_ | | **Inter-token tail (p95)** @ 75 tokens | 52.4 ms | **42.7 ms**  _(INT3)_ | | **Repeat-prompt TTFT** (KV cache hit) | ~160 ms | **4–11 ms** | | **Peak RAM** during inference | 5.14 GB | **3.50 GB** | | **Disk** — 7B INT4 / INT3 | 4.36 GB / — | **4.00 / 3.56 GB** | -| **Cold short-prompt TTFT** | **167 ms** | 192 ms  _(honest loss)_ | +| **Cold, unique-prompt TTFT** @ 75 tokens | 812 ms | **800 ms**  _(1.15× faster)_ | -Squish wins decode throughput, inter-token tail latency, full-response time, and RAM — biggest on long contexts, where its KV cache **reuses the prefill instead of re-running it**. INT3 adds ~18 % decode over INT4 at **no measured accuracy cost** (arc_easy `acc_norm` 0.551 vs 0.541, tied). The one place Ollama wins is single-token latency on a *cold, novel* prompt. +\* The 9.8× figure is the reuse ceiling — an identical prompt resent exactly. On completely unique prompts with 0% cache reuse, Squish is still faster across every context length tested, but by a smaller **1.15–1.32×** margin; see [`docs/paper.md` §4.4](docs/paper.md) for the floor-to-ceiling curve. + +Squish wins decode throughput, inter-token tail latency, full-response time, RAM, and TTFT — including on cold, unique prompts, where an earlier draft of these numbers had mistakenly credited Ollama a win (167 ms vs 192 ms) from a flawed same-fixed-prompt methodology that was actually measuring Ollama's own cache hit. Under genuinely cold, unique-prompt conditions Squish leads at every length tested. The gains are biggest on long, repeated contexts, where Squish's KV cache **reuses the prefill instead of re-running it**. INT3 adds ~18 % decode over INT4 at **no measured accuracy cost** (arc_easy `acc_norm` 0.551 vs 0.541, tied). → Methodology, thermal control, and the full ablation: [`docs/paper.md` §4.4](docs/paper.md) · [`BENCHMARKS.md`](BENCHMARKS.md) @@ -188,7 +190,6 @@ Honesty is a feature. If any of these matter, Ollama or LM Studio is the right c - **No GPU outside Apple Silicon.** It's MLX-based; CUDA users want vLLM or llama.cpp. - **No multi-user serving.** One developer, one machine — not a production API. - **No multimodal.** Text only. -- **Slower first token on a cold, short prompt** than Ollama (192 ms vs 167 ms) — fundamental MLX prefill kernel cost. Squish's edge is everywhere *else*. - **Model conversion is slow.** Squish needs models in its own format; first-time conversion takes minutes (`squish pull` skips it with pre-squished weights). ---