judge: {"match": "false"} is scored as PASS (bool("false") is True), and judge_llm defaults a missing verdict to pass

[reacher-z]

src/clawbench/runner/judge.py:187:

return bool(obj.get("match")), str(obj.get("reason", ""))

If a judge model replies {"match": "false", "reason": "..."} — a string, which LLMs emit routinely — then bool("false") is True and a mismatch is scored as a PASS. Verified: python3 -c 'print(bool("false"))' → True.

src/clawbench/runner/judge_llm.py:156 has the same bug plus a worse default:

return bool(obj.get("match", True)), str(obj.get("reason", ""))

A verdict object missing the match key defaults to True (pass). This is the lenient rubric — the one that produces the published Reward-lenient column.

Also conflated today: a missing/garbled verdict is indistinguishable from a genuine mismatch, both collapsing to False in judge.py.

Ask: parse tri-state explicitly — accept real booleans, accept case-insensitive "true"/"false" strings, return None (inconclusive) for anything else, and never default a missing key to a pass. _parse_verdict currently has no unit tests; a small matrix (bool / string / missing / garbage / nested) would pin this.

Impact is scoring integrity on published numbers, so worth doing before the next leaderboard refresh.

[reacher-z]

@Perry2004 — filed from a read-only audit of main; every claim is cited to file:line and was re-verified by hand before filing. Flagging for your triage: happy to send a PR for any of these you want picked up, or to close it if you judge the behaviour intentional.

[reacher-z]

Fix up in #305 (@Perry2004 requested as reviewer).

Reproduced on main first — {"match": "false"} and {"match": "FALSE"} both returned True from either judge, and judge_llm.py also returned True for a reply with no verdict key at all. _coerce_match() now maps real booleans and the common string spellings to a verdict and everything else to None, which run.py already treats as inconclusive (it gates on match is True). Adds a parse matrix over both modules; full suite 196 passed, ruff + pyright clean.

One thing this PR does not do: quantify how many published rows were affected. That needs a grep over archived judge outputs for stringified verdicts, then a targeted clawbench-rescore. Happy to take that as a follow-up if you want it before the next leaderboard refresh.