src/clawbench/runner/judge_llm.py:20-22 says it is a "drop-in replacement" for judge.py with the same interface. It has diverged in ways that matter, and it is the module behind the lenient rubric that produces the published Reward-lenient column:
|
judge.py |
judge_llm.py |
judge_context kwarg (passed at run.py:587-597) |
yes |
missing (:169-171) |
openai-responses api_type |
correct endpoint |
routed to /chat/completions (:180-181) |
google-generative-ai (the /v1beta/openai fix) |
supported (:99-108, :217-220) |
absent |
max_tokens |
4096, with a comment that reasoning judges burn hidden tokens (:123) |
800 (:114) |
The default judge is deepseek-v4-pro, a reasoning model — an 800-token budget risks truncating its verdict, and truncation lands on the except path whose lenient default is match=True (see #295). On top of that, _post_json / _build_user_msg / _call_* are ~100 duplicated lines that now have to be fixed twice; the Gemini fix already only landed in one copy.
Ask: reduce judge_llm.py to its JUDGE_SYSTEM prompt and delegate to judge.py's _run_judge / _build_user_msg with the rubric as a parameter. That restores identical api_type coverage, token budget, and signature in one place.
src/clawbench/runner/judge_llm.py:20-22says it is a "drop-in replacement" forjudge.pywith the same interface. It has diverged in ways that matter, and it is the module behind the lenient rubric that produces the published Reward-lenient column:judge.pyjudge_llm.pyjudge_contextkwarg (passed atrun.py:587-597):169-171)openai-responsesapi_type/chat/completions(:180-181)google-generative-ai(the/v1beta/openaifix):99-108,:217-220)max_tokens:123):114)The default judge is
deepseek-v4-pro, a reasoning model — an 800-token budget risks truncating its verdict, and truncation lands on theexceptpath whose lenient default ismatch=True(see #295). On top of that,_post_json/_build_user_msg/_call_*are ~100 duplicated lines that now have to be fixed twice; the Gemini fix already only landed in one copy.Ask: reduce
judge_llm.pyto itsJUDGE_SYSTEMprompt and delegate tojudge.py's_run_judge/_build_user_msgwith the rubric as a parameter. That restores identical api_type coverage, token budget, and signature in one place.