fix(rescore): resolve paths from the workspace instead of a maintainer's home

[vaibhavdabas16]

What does this PR do?

Fixes #296 — all four asks in the issue.

clawbench-rescore is an installed console script (pyproject.toml:29) that docs/scoring.md and eval/scoring.md both advertise as the way to reproduce leaderboard numbers from public traces. It shipped one machine's layout as its defaults:

--sweep-root   ~/work/ClawBench/claw-output/sweep
--models-yaml  ~/work/ClawBench/models/models.yaml

find_run_dirs() reaches the root through Path.rglob(), which yields nothing for a directory that does not exist rather than raising. So on any other machine the tool printed discovered 0 tasks and exited 0 — a silent no-op for precisely the audience the script exists to serve.

Ask	Change
1. Default --models-yaml to the workspace-resolved config	Now defaults to MODELS_YAML
2. Default --sweep-root to where runs land	Now WORKSPACE_ROOT / "test-output", matching clawbench-batch --output-dir (batch.py:833)
3. Error loudly instead of no-opping	A --sweep-root/--only-batch that isn't a directory, or a tree with no run-meta.json under it, prints a diagnostic naming the path and exits 2
4. Reuse load_model_config	The judge model is resolved through the shared loader instead of a private yaml.safe_load() + config.get("api_key")

Before and after, on a path that doesn't exist:

$ clawbench-rescore --sweep-root /definitely/not/here     # before
discovered 0 tasks, judging 0 intercepted ones ...
$ echo $?
0

$ clawbench-rescore --sweep-root /definitely/not/here     # after
ERROR: --sweep-root /definitely/not/here is not a directory.
       Point it at a tree containing completed runs (the runner writes them under ./test-output by default).
$ echo $?
2

One structural change worth calling out

Ask 4 could not be satisfied by importing run_support/config.py: that module runs ENGINE = _detect_engine() at import time (config.py:90) and sys.exit(1)s when neither Docker nor Podman is on PATH. rescore.py and both judges currently import none of it, so clawbench-rescore runs fine on a host with no container runtime — which is the normal case for someone scoring published traces. Importing config to reuse the loader would have handed a post-hoc scoring tool a hard Docker dependency, making the portability problem worse.

So load_models_yaml() and load_model_config() move to utils/model_config.py — they only ever needed WORKSPACE_ROOT and yaml — and both gain an optional explicit path argument. run_support/config.py re-exports both names (they were already in its __all__), so every existing importer, runner/run.py included, is untouched. A test asserts rescore never reaches for run_support.config again.

Happy to split that move into its own commit or drop it in favour of a narrower fix if you'd rather keep the loaders where they are.

Also extracted the argument parser into build_parser() so the shipped defaults are assertable from tests.

Corpus

• v2
• v1
• both
• not applicable

_{Host-side eval tooling; no task data involved.}

Test plan

• New tests/test_rescore_cli.py (10 tests): both defaults resolve under WORKSPACE_ROOT; a regression guard asserting no shipped default contains work/ClawBench; exit code 2 with a useful message for a missing root, an existing-but-empty root, and a missing --only-batch; load_model_config accepting both the api_keys list form and the api_key scalar form; the error naming the explicit --models-yaml path; and a guard that rescore does not import the container-probing module.
• Verified the old defaults really were Path.home() / "work/ClawBench/..." by reverting the file and re-reading them, and confirmed Path("/definitely/not/here").rglob("run-meta.json") returns [] rather than raising — the mechanism behind the silent no-op.
• Confirmed import clawbench.eval.rescore succeeds with shutil.which stubbed to return None for every command, i.e. with no container engine installed at all.
• Full suite: 202 passed, 3 skipped. The single failure, test_host_tasks.py::test_checked_task_json_files_parse_and_validate[v1-lite], reproduces identically on a clean main on this machine — the v1-lite task files are git symlinks (mode 120000) that Windows checks out as text. Unrelated to this change.
• ruff check and ruff format --check clean on all four files.

Related issues

Fixes #296.

_{Noticed while in here, left out of scope: batch.py:44-61 and tui.py:110/:346 each carry their own private copy of MODELS_YAML + load_models_yaml. Now that there is one shared home for them, those two could be collapsed onto it in a follow-up if you want it.}

[Perry2004]

At least one of the added test case is failing in CI. Please resolve this before further review, thanks.
https://github.com/TIGER-AI-Lab/ClawBench/actions/runs/32335738442/job/96648767983?pr=313

[vaibhavdabas16]

Pushed 431c409 to fix the red CI on this PR. The failure was in a test I added here, not in the fix itself.

What failed

tests/test_rescore_cli.py::test_defaults_do_not_ship_a_maintainers_home_layout failed on ubuntu-latest and macos-latest and passed on windows-latest:

AssertionError: assert 'work/ClawBench' not in '/home/runner/work/ClawBench/ClawBench/test-output'

The guard was written as a substring check against the resolved default paths:

for value in (args.sweep_root, args.models_yaml):
    assert "work/ClawBench" not in value.as_posix()

Actions checks this repo out to /home/runner/work/ClawBench/ClawBench, so a default that is correctly derived from WORKSPACE_ROOT still contains the substring work/ClawBench. The Windows runner passed only because its checkout path is D:\a\ClawBench\ClawBench. The assertion was testing the runner's directory layout, not the code.

What changed

The property actually worth guarding is that the defaults are not anchored to the invoking user's home directory. That cannot be read off the resolved paths from an arbitrary checkout location, so it is now asserted against the source of build_parser():

src = inspect.getsource(rescore.build_parser)

assert "Path.home()" not in src
assert "expanduser" not in src

test_defaults_are_workspace_relative already pins the resolved values exactly (WORKSPACE_ROOT / "test-output" and MODELS_YAML), so coverage of #296 is unchanged. The diff is test-only — no change to rescore.py, model_config.py, or config.py.

Verification

I reproduced the CI condition locally by forcing the workspace to a runner-shaped path via CLAWBENCH_WORKSPACE=.../home/runner/work/ClawBench/ClawBench:

	under a work/ClawBench workspace
old assertion	fails, with the same pytest diff as CI
new assertion	10 passed

The control matters: the old form fails under that path, so the reproduction is faithful rather than a vacuous pass.

Full suite locally: 202 passed, 3 skipped. The one failure, test_host_tasks.py::test_checked_task_json_files_parse_and_validate[v1-lite], is a local Windows artifact — those task files are git symlinks (mode 120000) that Windows checks out as text — and it is green on CI.

Current state

The workflow runs for the new head are sitting in action_required (0s, never executed), so the checks on this PR are pending a maintainer approving the run rather than failing. Happy to adjust the guard if you would prefer a different form.

[vaibhavdabas16]

Note for maintainers: this PR overlaps #314

Separate from the CI fix above, and only relevant at merge time.

This PR and #314 both touch src/clawbench/runner/run_support/config.py, from opposite directions:

• fix(rescore): resolve paths from the workspace instead of a maintainer's home #313 (this PR) moves load_models_yaml() and load_model_config() out to utils/model_config.py and leaves re-exports behind, so that eval/rescore.py can reuse them. It cannot import run_support/config.py directly: that module resolves a container engine at import time and exits when neither Docker nor Podman is present, which would make a post-hoc scoring tool require a container runtime.
• fix(judge): a bad --judge model no longer discards a completed run #314 rewrites those same two functions in place, so they raise ModelConfigError instead of calling sys.exit.

Each merges cleanly into main on its own. Verified against the current heads (431c409 and 2c777ab) with git merge-tree:

313 into main  -> clean
314 into main  -> clean
313 vs 314     -> CONFLICT (content): src/clawbench/runner/run_support/config.py

So two things can happen, depending on merge order:

1. fix(rescore): resolve paths from the workspace instead of a maintainer's home #313 lands first — fix(judge): a bad --judge model no longer discards a completed run #314 then needs a rebase, and its ModelConfigError change should be applied to the moved functions in utils/model_config.py rather than to run_support/config.py. The re-exports mean nothing else has to move.
2. fix(judge): a bad --judge model no longer discards a completed run #314 lands first — fix(rescore): resolve paths from the workspace instead of a maintainer's home #313 then needs a rebase, carrying fix(judge): a bad --judge model no longer discards a completed run #314's ModelConfigError bodies through the move into utils/model_config.py.

Either order works and neither is blocked; only the second one to merge needs the rebase. No preference from me — say which you would rather take first and I will rebase the other onto it promptly.

One thing worth a maintainer's eye either way: after both land, load_model_config() will raise ModelConfigError in a path that rescore calls. This PR's test_load_model_config_reports_the_explicit_path_on_a_bad_model currently asserts SystemExit, and that assertion will need to change to match whichever behaviour ends up canonical.