Skip to content

alternate port: gpt-5.6-sol-pro — 37/37 effects, $30.07 vs $43, retrieval instead of context - #13

Closed
staccDOTsol wants to merge 2 commits into
omacom:masterfrom
staccDOTsol:zoo/sol
Closed

alternate port: gpt-5.6-sol-pro — 37/37 effects, $30.07 vs $43, retrieval instead of context#13
staccDOTsol wants to merge 2 commits into
omacom:masterfrom
staccDOTsol:zoo/sol

Conversation

@staccDOTsol

@staccDOTsol staccDOTsol commented Aug 16, 2026

Copy link
Copy Markdown

alternate port: openai/gpt-5.6-sol-pro — $30.2842 all-in vs $43 DHH paid for the same model (beats by 29.6%)

One of five frontier models given DHH's exact challenge (same plan.md, same
Python reference) with one change: the 396k-token reference was retrieved
per-ask
through a holographic-memory layer instead of re-sent in context.

  • 37/37 effects, each gated on compiling AND rendering styled ANSI animation
  • $30.2842 total, provider-billed, every ask on the attached LEDGER.jsonl
  • 8.6x measured vs carrying the corpus (real cold/warm sends, not assumed cache rates)
  • watch the effects run from this binary: https://ttfx.awesomemcp.fun/show/sol

Parity, measured on this repo's own pinned platform (Linux/glibc 2.35), against
this repo's own fixtures (tests/fixtures/easing_goldens.bin):

13/13 implemented easing functions, 13,013/13,013 samples — 100.00% bit-exact.

Two asks got there, both on the attached ledger: the original port, then a $0.21
follow-up after our bridge flagged a 1-ulp expression-order gap (CPython's
x ** 3 routes through libm pow(); sol rewrote its cubics to .powf(3.0)).
The full 354-case suite remains this repository's bar — nothing beyond the
measured subset is claimed. This PR adds alternates/openai_gpt-5_6-sol-pro/:
the same challenge on the cost axis, receipts included.

honesty section (condensed from the run's live during-mortem)

The harness was improved WHILE this ran, and every dollar of that is in the
numbers above, not edited out: a render gate added mid-run dropped and re-bought
already-paid effects; three reasoning-budget truncation bugs bought ~$7 of
unusable responses before diagnosis; early asks shipped 3x-fat retrieval chunks;
restarts re-bought cores. Projections use the blended all-waste rate (marginal
rates run ~5x lower). The "without retrieval" comparison is MEASURED, not
modeled — real cold+warm full-corpus sends at provider-billed prices, which
found NO default cache discount for Anthropic or DeepSeek-via-OpenRouter. DHH's
side counts only his published successes; ours counts every failure. The
finished Rust port in this repo was quarantined from the models throughout —
they saw only plan.md and the Python source. Full ledger of every ask attached
(LEDGER.jsonl: tokens, provider-billed USD, timestamps).

Close it if exhibits don't belong here — the branch and the live page stand on
their own either way.

… via leCore retrieval

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… platform)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@dhh

dhh commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Very cool experiment, but let's setup a separate repo for this, so we don't pollute the production PRs for ttfx. Do post where you set it up!

@dhh dhh closed this Aug 17, 2026
@staccDOTsol

Copy link
Copy Markdown
Author

Set it up here: https://github.com/staccDOTsol/ttfx-zoo

All five model ports live on `master` with their provider-billed ledgers, and the original per-model branches are preserved.

model effects cost you paid
deepseek-v4-pro 37/37 $4.4898 $23
grok-4.6 37/37 $10.4965 $55
claude-sonnet-5 37/37 $17.2118
gpt-5.6-sol-pro 37/37 $30.2842 $43
claude-fable-5 37/37 $54.9769 $550

Same `plan.md`, same Python reference, same gate (compiles AND renders styled ANSI) — the only change is that the 396k-token reference was retrieved per ask instead of re-sent in context. Totals are recomputed from the `LEDGER.jsonl` files rather than copied from the PR titles.

Caveats are in the README rather than buried: ask counts vary 50–96 because retries are included, grok-4.6's ledger carries waste from a period its provider key was being rejected, your baselines are your reported figures rather than re-measured, and sonnet-5 has no baseline because you didn't run it. Also stated plainly: this only helps when the reference is much larger than the ask — for small references, retrieval costs more than just sending it.

Thanks for the nudge to split it out.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants