fix(judge): {"match": "false"} is scored as a PASS — parse verdicts as tri-state (closes #295)

[reacher-z]

Closes #295. Scoring-correctness fix — it changes numbers, so it deserves a careful read.

The bug

Both judges turned the model's verdict into a boolean with bool():

# judge.py:187 (strict rubric)
return bool(obj.get("match")), str(obj.get("reason", ""))
# judge_llm.py:156 (lenient rubric — produces the published Reward-lenient column)
return bool(obj.get("match", True)), str(obj.get("reason", ""))

bool("false") is True. A judge replying {"match": "false"} — a string, which models emit routinely — had its mismatch recorded as a PASS. Reproduced on main before the fix:

judge reply	judge.py	judge_llm.py
{"match": "false", "reason": "wrong item"}	True ❌	True ❌
{"match": "FALSE"}	True ❌	True ❌
{"reason": "forgot the verdict"}	False	True ❌
{"match": false}	False ✓	False ✓

judge_llm.py was worse on two counts: a missing match key defaulted to True, and its keyword fallback returned True for any reply it could not parse. Both inflate the lenient column specifically.

The fix

_coerce_match() maps real booleans and the common string spellings ("true"/"false"/"yes"/"no"/"pass"/"fail"/"match"/"mismatch", case-insensitive) to a verdict, and everything else — missing key, null, "maybe", junk — to None.

None is not a new state: judge.py:248/267 already returns it for call failures, and run.py gates on judge_result.get("match") is True (:662, :746), so an inconclusive verdict is not a pass. The lenient keyword fallback now returns None instead of inventing a pass.

After the fix, every row above resolves correctly ("false"/"FALSE" → False, missing key → None).

Tests

New parse matrix over both modules — booleans, string spellings, casing, fenced JSON, missing key, null, non-verdict junk, and a type-rejection case for _coerce_match. Full suite: 196 passed; ruff and pyright clean.

What this means for published numbers

Any run whose judge answered with a stringly-typed mismatch was scored as a pass. I have not quantified how many rows that affects — it depends on how often each judge model stringifies, which is visible in the stored raw verdicts. Suggested follow-up: grep the archived judge outputs for "match": " and, if there are hits, re-run clawbench-rescore for the affected rows before the next leaderboard refresh. Happy to do that as a separate PR.

@Perry2004 — please review carefully: this changes what counts as a pass. The two calls I'd most like a second opinion on are (1) treating a missing verdict key as inconclusive rather than a mismatch, and (2) the accepted string vocabulary — I kept it deliberately narrow.

[Perry2004]

I have checked that the bug only happens when the judge LLM returns a malformed result (returning falsy strings/texts instead of valid boolean). Therefore it wont influence the validity of existing judges.

The fix LGTM. Will merge and include in a patch release.