Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added docs/assets/blog/squish-building.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/blog/squish-fistpump.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/blog/squish-flying.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/blog/squish-hero.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
619 changes: 589 additions & 30 deletions docs/blog/posts/local-llm-fast-enough.md

Large diffs are not rendered by default.

4 changes: 2 additions & 2 deletions docs/paper.md
Original file line number Diff line number Diff line change
Expand Up @@ -153,7 +153,7 @@ This section reports end-to-end, on-hardware throughput and latency — the live

Per-token decode (median inter-token latency) is 39–49 ms for Squish versus 50–57 ms for Ollama across all lengths; tail latency (p95) is uniformly tighter. These gains are *serving-layer* recoveries, not model differences: decoupling the decode loop from the event loop (one inference-thread handoff per request rather than per token), suspending cyclic garbage collection during generation with a one-time post-warmup heap freeze, and pinning the inference thread to performance cores via QoS together raised short-context throughput from 14.6 to 20.5 tok/s and cut the p95 tail from 166 ms to 48 ms.

**Long-context end-to-end.** The largest gap is full-response latency on long prompts. For a 200-token completion on a 4 000-token prompt, Ollama takes 37.5 s versus Squish's 3.8 s — a **9.8×** reduction — because Squish's KV cache reuses the prefill across requests where Ollama re-runs it. On a genuinely novel 4 000-token prompt (caches cold for both), prefill is compute-bound and the two engines are at parity (~26 s); Squish's advantage is architectural cache reuse, not faster cold prefill.
**Long-context end-to-end.** The largest gap is full-response latency on long prompts. For a 200-token completion on a 4 000-token prompt, Ollama takes 37.5 s versus Squish's 3.8 s — a **9.8×** reduction — because Squish's KV cache reuses the prefill across requests where Ollama re-runs it. On a genuinely novel 4 000-token prompt (caches cold for both, 0% reuse), prefill is compute-bound and Squish still leads, by a genuine 1.15–1.32× margin across tested context lengths rather than the cache-reuse ceiling above; Squish's advantage there comes from serving-layer throughput, not cache reuse.

**Decode-acceleration ablation.** Decode on this hardware is *weight-bandwidth bound*: each token streams the full ~4 GB INT4 weight set from unified memory, giving a ~25 tok/s ceiling of which Squish reaches ~20–22. Two levers move that ceiling and two do not:

Expand All @@ -162,7 +162,7 @@ Per-token decode (median inter-token latency) is 39–49 ms for Squish versus 50
- **Quantised KV cache** (mlx_lm-native, GPU-side): **no decode speedup** at 4- or 8-bit (20.0 tok/s unchanged), because the KV cache is only ~10–15 % of per-token memory traffic and the dequantisation-during-attention cost offsets the bandwidth saved; 4-bit additionally degrades output. It is a *memory* lever (longer context in fixed RAM), not a throughput lever.
- **Draft-model speculative decoding** (1.5 B draft for the 7 B target): net-negative at temperature 0 — acceptance is too low to offset the draft forwards, and the path was retained as opt-in only.

The throughline is that on bandwidth-bound Apple-Silicon decode, the techniques that help are those that read fewer weight bytes per token (lower-bit quantisation) or emit more tokens per weight read (greedy-lossless speculation); techniques that compress only the KV cache or add a draft model do not. Squish's only measured loss against Ollama is time-to-first-token on a cold short prompt (192 ms vs 167 ms); it leads on decode throughput, tail latency, full-response latency, and RAM (3.5 GB vs 5.1 GB peak).
The throughline is that on bandwidth-bound Apple-Silicon decode, the techniques that help are those that read fewer weight bytes per token (lower-bit quantisation) or emit more tokens per weight read (greedy-lossless speculation); techniques that compress only the KV cache or add a draft model do not. Squish has no measured loss against Ollama: under cold, genuinely unique-prompt conditions it leads time-to-first-token at every length tested, down to 75 tokens, in addition to decode throughput, tail latency, full-response latency, and RAM (3.5 GB vs 5.1 GB peak).

---

Expand Down
10 changes: 9 additions & 1 deletion overrides/main.html
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,15 @@
<meta property="og:image:alt" content="{{ og_title }}">

{% if og_type == "article" %}
<meta property="article:published_time" content="{{ page.meta.date }}">
{#- `date` is either a flat value or a {created, updated} mapping. -#}
{% if page.meta.date is mapping %}
<meta property="article:published_time" content="{{ page.meta.date.created }}">
{% if page.meta.date.updated %}
<meta property="article:modified_time" content="{{ page.meta.date.updated }}">
{% endif %}
{% else %}
<meta property="article:published_time" content="{{ page.meta.date }}">
{% endif %}
{% if page.meta.og_author %}
<meta property="article:author" content="{{ page.meta.og_author }}">
{% endif %}
Expand Down
Loading