Skip to content

docs(readme): fix stale TTFT claim, clarify reuse-ceiling framing#223

Merged
wesleyscholl merged 1 commit into
mainfrom
claude/squish-blog-sync-deploy-8ed169
Jul 7, 2026
Merged

docs(readme): fix stale TTFT claim, clarify reuse-ceiling framing#223
wesleyscholl merged 1 commit into
mainfrom
claude/squish-blog-sync-deploy-8ed169

Conversation

@konjoinfinity

Copy link
Copy Markdown
Collaborator

Summary

Syncs the README's benchmark table against the now-corrected numbers in the blog post (docs/blog/posts/local-llm-fast-enough.md) and docs/paper.md §4.4, which this session already fixed after the same stale claim was caught there.

Two stale claims found and fixed:

  • "Cold short-prompt TTFT: 167ms vs 192ms, Squish loses." This came from a flawed methodology (the same fixed prompt sent five times), which was actually measuring Ollama's own cache hit on the repeat sends, not a genuinely cold comparison. Replaced with the corrected cold/unique-prompt floor benchmark: Squish leads 800ms vs 812ms at 75 tokens (1.15× faster) — Squish has no measured TTFT loss anywhere under real cold, unique-prompt conditions.
  • The same claim repeated in "What Squish Doesn't Do" as a listed limitation ("Slower first token on a cold, short prompt than Ollama"). Removed entirely, since it's no longer true.

One clarification added (not a correction, but avoids reproducing the same misleading framing the blog post's original stale draft had):

  • The "Full response @ 4000-token prompt" row shows the 9.8× reuse ceiling (an identical prompt resent exactly) without qualification. Added (repeated exactly) to the row label and a footnote pointing to the real unique-prompt floor (1.15–1.32×), matching how the blog post now presents both numbers side by side.

Also found, out of scope for this PR: BENCHMARKS.md has the same stale claim in three different variants (lines 24, 70, 74 — including a third, different pair of numbers: 131ms vs 279ms). Flagging this as a separate follow-up needed since the ask here was specifically the README.

Type of change

  • Bug fix
  • New feature
  • Performance improvement
  • Documentation update
  • Refactor / cleanup

Checklist

  • No hardcoded absolute paths
  • No model weights, eval output files, or log files staged for commit
  • Changes are scoped to one logical concern (README benchmark accuracy)
  • pytest tests/ passes locally — N/A, docs-only change
  • ruff check squish/ tests/ reports no errors — N/A, no Python changed
  • Performance-sensitive changes include a before/after squish bench run — N/A

Related issues

Follow-up to #215#222 (the blog post sync/correction work this stale claim was originally caught in).


Generated by Claude Code

Two stale claims carried over from the same disproven benchmark the
blog post and paper.md already corrected:

- The "Cold short-prompt TTFT" row claimed Ollama beat Squish 167ms vs
  192ms. That number came from a flawed same-fixed-prompt-repeated-5x
  methodology that was actually measuring Ollama's own cache hit, not
  a genuinely cold comparison. Replaced with the corrected cold/unique
  floor benchmark: Squish leads 800ms vs 812ms at 75 tokens.
- The "What Squish Doesn't Do" section repeated the same disproven
  167/192 claim as a limitation. Removed — Squish has no measured TTFT
  loss under real cold, unique-prompt conditions.
- Added a footnote clarifying the 9.8x figure is the exact-repeat reuse
  ceiling, not the typical case, and pointed to the actual 1.15-1.32x
  floor for unique prompts.
@wesleyscholl wesleyscholl marked this pull request as ready for review July 7, 2026 11:29
@wesleyscholl wesleyscholl merged commit 388a709 into main Jul 7, 2026
17 checks passed
@wesleyscholl wesleyscholl deleted the claude/squish-blog-sync-deploy-8ed169 branch July 7, 2026 11:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants