Skip to content

batch --resume skips failed jobs (log-file heuristic) and overwrites batch-summary.json with 'skipped' rows #297

Description

@reacher-z

src/clawbench/runner/batch.py:651-655:

for job in jobs:
    safe_model = re.sub(r"[/:]+", "--", job.model)
    if (log_dir / f"{job.case_name}-{safe_model}.log").exists():
        job.status = "skipped"

The log file is written unconditionally when a job starts (batch.py:320), so --resume skips every job that was ever attempted — including the ones that failed or errored. That inverts the flag's main use case: after a network blip or a container OOM kills 20 of 130 tasks, --resume skips exactly those 20 and reports the batch complete.

Second effect: after resume, write_summary_json (batch.py:550-579, called at :754) rewrites batch-summary.json from the in-memory job list, so every previously-completed job is recorded as status: "skipped", duration_seconds: 0. The original tallies are destroyed in the artifact that downstream stats and the HF upload consume.

Ask:

  1. On resume, decide completion from each run's run-meta.json (authoritative outcome), not from log-file existence.
  2. Add --retry-failed to re-run jobs whose recorded outcome was failure/infra error.
  3. Merge prior job statuses into the rewritten summary rather than overwriting them with skipped.

Related: #160 (resume/resilience testing) asks for exactly this path to be verified end-to-end; there is currently no test covering resume (tests/test_batch_and_tui_helpers.py only passes resume=None).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions