docs(readme): fix stale TTFT claim, clarify reuse-ceiling framing#223
Merged
Conversation
Two stale claims carried over from the same disproven benchmark the blog post and paper.md already corrected: - The "Cold short-prompt TTFT" row claimed Ollama beat Squish 167ms vs 192ms. That number came from a flawed same-fixed-prompt-repeated-5x methodology that was actually measuring Ollama's own cache hit, not a genuinely cold comparison. Replaced with the corrected cold/unique floor benchmark: Squish leads 800ms vs 812ms at 75 tokens. - The "What Squish Doesn't Do" section repeated the same disproven 167/192 claim as a limitation. Removed — Squish has no measured TTFT loss under real cold, unique-prompt conditions. - Added a footnote clarifying the 9.8x figure is the exact-repeat reuse ceiling, not the typical case, and pointed to the actual 1.15-1.32x floor for unique prompts.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Syncs the README's benchmark table against the now-corrected numbers in the blog post (
docs/blog/posts/local-llm-fast-enough.md) anddocs/paper.md§4.4, which this session already fixed after the same stale claim was caught there.Two stale claims found and fixed:
One clarification added (not a correction, but avoids reproducing the same misleading framing the blog post's original stale draft had):
(repeated exactly)to the row label and a footnote pointing to the real unique-prompt floor (1.15–1.32×), matching how the blog post now presents both numbers side by side.Also found, out of scope for this PR:
BENCHMARKS.mdhas the same stale claim in three different variants (lines 24, 70, 74 — including a third, different pair of numbers: 131ms vs 279ms). Flagging this as a separate follow-up needed since the ask here was specifically the README.Type of change
Checklist
pytest tests/passes locally — N/A, docs-only changeruff check squish/ tests/reports no errors — N/A, no Python changedsquish benchrun — N/ARelated issues
Follow-up to #215–#222 (the blog post sync/correction work this stale claim was originally caught in).
Generated by Claude Code