src/clawbench/runner/run.py:578 loads the judge config after the agent run finishes, inside the judge stage:
judge_cfg = load_model_config(args.judge)
But load_model_config calls sys.exit(1) on a missing model (run_support/config.py:164-166, also :180, :191, :200). SystemExit derives from BaseException, so it escapes both the judge-stage except Exception (run.py:606) and the outer except Exception (run.py:674) — and write_run_meta (run.py:665/:722) never runs.
Net effect for a first-time user whose models.yaml lacks deepseek-v4-pro (the default judge — models.example.yaml even warns it "must exist here or the scoring stage fails"): they burn a full agent run of up to 30 minutes, the interception may well have succeeded, and then the process dies leaving no run-meta.json at all. The run is invisible to batch stats, to clawbench-rescore (which globs run-meta.json), and to HF upload.
Ask:
- Validate the judge config at startup, before the agent run — next to the existing
preflight_model_api call at run.py:352.
- Make
load_model_config raise a catchable exception rather than sys.exit, or catch SystemExit in the judge stage and degrade to a judge_setup_failed outcome that still writes run-meta.json.
Either way the invariant worth stating: a completed agent run always writes its metadata, whatever happens at scoring time.
src/clawbench/runner/run.py:578loads the judge config after the agent run finishes, inside the judge stage:But
load_model_configcallssys.exit(1)on a missing model (run_support/config.py:164-166, also:180,:191,:200).SystemExitderives fromBaseException, so it escapes both the judge-stageexcept Exception(run.py:606) and the outerexcept Exception(run.py:674) — andwrite_run_meta(run.py:665/:722) never runs.Net effect for a first-time user whose
models.yamllacksdeepseek-v4-pro(the default judge —models.example.yamleven warns it "must exist here or the scoring stage fails"): they burn a full agent run of up to 30 minutes, the interception may well have succeeded, and then the process dies leaving norun-meta.jsonat all. The run is invisible to batch stats, toclawbench-rescore(which globsrun-meta.json), and to HF upload.Ask:
preflight_model_apicall atrun.py:352.load_model_configraise a catchable exception rather thansys.exit, or catchSystemExitin the judge stage and degrade to ajudge_setup_failedoutcome that still writesrun-meta.json.Either way the invariant worth stating: a completed agent run always writes its metadata, whatever happens at scoring time.