alternate port: gpt-5.6-sol-pro — 37/37 effects, $30.07 vs $43, retrieval instead of context - #13
alternate port: gpt-5.6-sol-pro — 37/37 effects, $30.07 vs $43, retrieval instead of context#13staccDOTsol wants to merge 2 commits into
Conversation
… via leCore retrieval Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… platform) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Very cool experiment, but let's setup a separate repo for this, so we don't pollute the production PRs for ttfx. Do post where you set it up! |
|
Set it up here: https://github.com/staccDOTsol/ttfx-zoo All five model ports live on `master` with their provider-billed ledgers, and the original per-model branches are preserved.
Same `plan.md`, same Python reference, same gate (compiles AND renders styled ANSI) — the only change is that the 396k-token reference was retrieved per ask instead of re-sent in context. Totals are recomputed from the `LEDGER.jsonl` files rather than copied from the PR titles. Caveats are in the README rather than buried: ask counts vary 50–96 because retries are included, grok-4.6's ledger carries waste from a period its provider key was being rejected, your baselines are your reported figures rather than re-measured, and sonnet-5 has no baseline because you didn't run it. Also stated plainly: this only helps when the reference is much larger than the ask — for small references, retrieval costs more than just sending it. Thanks for the nudge to split it out. |
alternate port: openai/gpt-5.6-sol-pro — $30.2842 all-in vs $43 DHH paid for the same model (beats by 29.6%)
One of five frontier models given DHH's exact challenge (same plan.md, same
Python reference) with one change: the 396k-token reference was retrieved
per-ask through a holographic-memory layer instead of re-sent in context.
Parity, measured on this repo's own pinned platform (Linux/glibc 2.35), against
this repo's own fixtures (
tests/fixtures/easing_goldens.bin):13/13 implemented easing functions, 13,013/13,013 samples — 100.00% bit-exact.
Two asks got there, both on the attached ledger: the original port, then a $0.21
follow-up after our bridge flagged a 1-ulp expression-order gap (CPython's
x ** 3routes through libm pow(); sol rewrote its cubics to.powf(3.0)).The full 354-case suite remains this repository's bar — nothing beyond the
measured subset is claimed. This PR adds
alternates/openai_gpt-5_6-sol-pro/:the same challenge on the cost axis, receipts included.
honesty section (condensed from the run's live during-mortem)
The harness was improved WHILE this ran, and every dollar of that is in the
numbers above, not edited out: a render gate added mid-run dropped and re-bought
already-paid effects; three reasoning-budget truncation bugs bought ~$7 of
unusable responses before diagnosis; early asks shipped 3x-fat retrieval chunks;
restarts re-bought cores. Projections use the blended all-waste rate (marginal
rates run ~5x lower). The "without retrieval" comparison is MEASURED, not
modeled — real cold+warm full-corpus sends at provider-billed prices, which
found NO default cache discount for Anthropic or DeepSeek-via-OpenRouter. DHH's
side counts only his published successes; ours counts every failure. The
finished Rust port in this repo was quarantined from the models throughout —
they saw only plan.md and the Python source. Full ledger of every ask attached
(LEDGER.jsonl: tokens, provider-billed USD, timestamps).
Close it if exhibits don't belong here — the branch and the live page stand on
their own either way.