Skip to content

A missing judge model kills the process after the agent run and loses run-meta.json entirely (SystemExit escapes both handlers) #302

Description

@reacher-z

src/clawbench/runner/run.py:578 loads the judge config after the agent run finishes, inside the judge stage:

judge_cfg = load_model_config(args.judge)

But load_model_config calls sys.exit(1) on a missing model (run_support/config.py:164-166, also :180, :191, :200). SystemExit derives from BaseException, so it escapes both the judge-stage except Exception (run.py:606) and the outer except Exception (run.py:674) — and write_run_meta (run.py:665/:722) never runs.

Net effect for a first-time user whose models.yaml lacks deepseek-v4-pro (the default judge — models.example.yaml even warns it "must exist here or the scoring stage fails"): they burn a full agent run of up to 30 minutes, the interception may well have succeeded, and then the process dies leaving no run-meta.json at all. The run is invisible to batch stats, to clawbench-rescore (which globs run-meta.json), and to HF upload.

Ask:

  1. Validate the judge config at startup, before the agent run — next to the existing preflight_model_api call at run.py:352.
  2. Make load_model_config raise a catchable exception rather than sys.exit, or catch SystemExit in the judge stage and degrade to a judge_setup_failed outcome that still writes run-meta.json.

Either way the invariant worth stating: a completed agent run always writes its metadata, whatever happens at scoring time.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions