Skip to content

judge_llm.py has diverged from judge.py (no Gemini support, wrong responses endpoint, 800-token cap) — collapse the duplication #300

Description

@reacher-z

src/clawbench/runner/judge_llm.py:20-22 says it is a "drop-in replacement" for judge.py with the same interface. It has diverged in ways that matter, and it is the module behind the lenient rubric that produces the published Reward-lenient column:

judge.py judge_llm.py
judge_context kwarg (passed at run.py:587-597) yes missing (:169-171)
openai-responses api_type correct endpoint routed to /chat/completions (:180-181)
google-generative-ai (the /v1beta/openai fix) supported (:99-108, :217-220) absent
max_tokens 4096, with a comment that reasoning judges burn hidden tokens (:123) 800 (:114)

The default judge is deepseek-v4-pro, a reasoning model — an 800-token budget risks truncating its verdict, and truncation lands on the except path whose lenient default is match=True (see #295). On top of that, _post_json / _build_user_msg / _call_* are ~100 duplicated lines that now have to be fixed twice; the Gemini fix already only landed in one copy.

Ask: reduce judge_llm.py to its JUDGE_SYSTEM prompt and delegate to judge.py's _run_judge / _build_user_msg with the rubric as a parameter. That restores identical api_type coverage, token budget, and signature in one place.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions