Almost 100%

[moro-no-kimi]

What the blog reports vs. what this artifact actually is

From the blog (Agent swarms and the new model economics):

"The Opus mix shows the same shape with 19,013 lines at a 97% grade under the old harness, and 4,645 lines at 100% under the new harness."
"…both the old and new swarms ultimately passed the full suite…"

Upon further reading, is it correct that this repo is the codebase from the solo Opus run? About which the blog explicitly states:

"To get a sense of solo frontier costs, we also ran Opus … We graded those runs only informally, so we draw no conclusions about their quality here."

If that's correct, then this published artifact never received the same held-out, cross-checked grading the blog's "100%" refers to. For that reason, I decided to close that gap by running the sqllogictest corpus against it.

What I measured against the standard sqllogictest corpus

Corpus: SQLite's own sqllogictest mirror, gregrahn/sqllogictest @ pinned commit c67f97bf3ca7e590d12e073408bcacaf2ff0f3a0.

Driver: minisqlite is library-only, so a ~40-line Rust shim exposes Connection::{execute, query} over the standard test-runner contract (binary exec → stdin SQL; statement success prints OK; queries print one value per line; errors print ERROR <CODE> to stderr / exit 1). Per-file scratch working directory → each test gets a fresh database. Each .test file is fed to the shim in exec mode, matching how an independent runner drives any engine.

Overall result

=== 4,704,391 pass / 3,249 fail / 1,226,141 skip across 497 files ===

99.93% pass rate on graded records — very strong, and not 100%.

Corpus region	files	pass	fail	skip	pass rate
index/random/*	24	192,850	3,220	38,920	98.36%
everything else	473	4,511,541	29	1,187,221	100%
total	497	4,704,391	3,249	1,226,141	99.931%

The three failure classes

1. index/random/* correlated queries — the dominant class (3,220 records across 24 files)

For these shapes minisqlite's query planner computes a different result set than SQLite (hash mismatch on multi-row results). Representative pattern from index/random/10/slt_good_*.test:

SELECT col2 col1 FROM tab3 AS cor0 WHERE ( NULL ) > NULL;

This is the substantive finding: indexed correlated-subquery / pruning combinations diverge from the reference engine. It is a genuine planner/executor divergence, not a formatting artifact.

2. Integer division with a CAST(... AS REAL) operand — ~8 records

SELECT - 42 / CAST( + - 96 AS REAL ) + + - 46;

Both minisqlite and reference SQLite return -45.5625; the corpus's expected -45 reflects the sqllogictest reference harness's query I type-tag leniency (results coerced to integers under that tag). Our strict runner doesn't apply the type-tag coercion and flags it. Consider this "harness stricter than sqllogictest," not an engine bug.

3. Floating-point zero sign — ~8 records

Your engine preserves IEEE negative zero (Real(-0) → -0); SQLite's %!.15g formatter normalizes it to 0. Cosmetic, but counted by a byte-exact grader.

Asks

1. A one-line README note clarifying that this published solo-Opus artifact has not been independently scored against the standard sqllogictest corpus.
2. If it's useful, I'm happy to share the exact runner + shim source so the team can verify. Ideally the team could add their own reproducible sqllogictest script to this repo.