src/clawbench/runner/batch.py:651-655:
for job in jobs:
safe_model = re.sub(r"[/:]+", "--", job.model)
if (log_dir / f"{job.case_name}-{safe_model}.log").exists():
job.status = "skipped"
The log file is written unconditionally when a job starts (batch.py:320), so --resume skips every job that was ever attempted — including the ones that failed or errored. That inverts the flag's main use case: after a network blip or a container OOM kills 20 of 130 tasks, --resume skips exactly those 20 and reports the batch complete.
Second effect: after resume, write_summary_json (batch.py:550-579, called at :754) rewrites batch-summary.json from the in-memory job list, so every previously-completed job is recorded as status: "skipped", duration_seconds: 0. The original tallies are destroyed in the artifact that downstream stats and the HF upload consume.
Ask:
- On resume, decide completion from each run's
run-meta.json (authoritative outcome), not from log-file existence.
- Add
--retry-failed to re-run jobs whose recorded outcome was failure/infra error.
- Merge prior job statuses into the rewritten summary rather than overwriting them with
skipped.
Related: #160 (resume/resilience testing) asks for exactly this path to be verified end-to-end; there is currently no test covering resume (tests/test_batch_and_tui_helpers.py only passes resume=None).
src/clawbench/runner/batch.py:651-655:The log file is written unconditionally when a job starts (
batch.py:320), so--resumeskips every job that was ever attempted — including the ones that failed or errored. That inverts the flag's main use case: after a network blip or a container OOM kills 20 of 130 tasks,--resumeskips exactly those 20 and reports the batch complete.Second effect: after resume,
write_summary_json(batch.py:550-579, called at:754) rewritesbatch-summary.jsonfrom the in-memory job list, so every previously-completed job is recorded asstatus: "skipped", duration_seconds: 0. The original tallies are destroyed in the artifact that downstream stats and the HF upload consume.Ask:
run-meta.json(authoritative outcome), not from log-file existence.--retry-failedto re-run jobs whose recorded outcome was failure/infra error.skipped.Related: #160 (resume/resilience testing) asks for exactly this path to be verified end-to-end; there is currently no test covering resume (
tests/test_batch_and_tui_helpers.pyonly passesresume=None).