Skip to content

fix(batch): one truncated run-meta.json no longer discards the batch summary - #312

Open
vaibhavdabas16 wants to merge 1 commit into
TIGER-AI-Lab:mainfrom
vaibhavdabas16:fix/batch-stats-atomic-run-meta
Open

fix(batch): one truncated run-meta.json no longer discards the batch summary#312
vaibhavdabas16 wants to merge 1 commit into
TIGER-AI-Lab:mainfrom
vaibhavdabas16:fix/batch-stats-atomic-run-meta

Conversation

@vaibhavdabas16

Copy link
Copy Markdown

What does this PR do?

Fixes #303: a single truncated run-meta.json currently destroys the machine-readable results for an entire batch, and the truncation itself is made impossible in the first place.

The crash. print_run_stats() parsed every run-meta.json with a bare json.loads() (batch.py:450). async_main() calls it at batch.py:753, one line before write_summary_json() at :754 — so one malformed file raised JSONDecodeError out of the function and took batch-summary.json down with it. Every other job in the batch lost its recorded results, even though those runs succeeded.

The truncation. write_run_meta() (metadata.py:288) wrote with a plain write_text(). A run killed mid-write — Ctrl-C, OOM, host shutdown — leaves a half-written file on disk, which is precisely what the reader above then chokes on.

Both halves are addressed:

  • New clawbench.utils.jsonio with write_json_atomic() (same-directory temp file, flush + fsync, then os.replace) and read_json_or_none(). os.replace replaces atomically on POSIX and Windows, so an interrupted write leaves either the previous file or no file — never a truncated one.
  • write_run_meta() and write_summary_json() now write atomically.
  • print_run_stats() reads via read_json_or_none(). It also rejects valid JSON that isn't an object — a bare list previously reached .get() and raised AttributeError — prints a warning naming the offending file, and falls back to the directory name for that one run.

Reporting now degrades to a single warning line for the affected run instead of losing the whole batch.

Corpus

  • v2
  • v1
  • both
  • not applicable

Host-side runner/reporting change; no task data involved.

Test plan

  • New tests/test_batch_stats_resilience.py (11 tests): atomic round-trip and parent-dir creation, no leftover temp files, previous file survives a failed serialization, read_json_or_none() against truncated / empty / garbage / undecodable input, and two print_run_stats() regression tests covering a truncated run-meta.json and a non-object one.
  • Confirmed the regression tests genuinely catch the bug: reverted batch.py and metadata.py to main and re-ran, which fails with JSONDecodeError and AttributeError respectively; both pass with the fix applied.
  • Full suite: 203 passed, 3 skipped. The one unrelated failure, test_host_tasks.py::test_checked_task_json_files_parse_and_validate[v1-lite], reproduces identically on a clean main checkout on this machine — the v1-lite task files are git symlinks (mode 120000) that Windows checks out as plain text.
  • ruff check and ruff format --check clean on all changed files. Removing the last json. uses from batch.py left import json unused, so it is dropped (F401).

Related issues

Fixes #303.

One note for maintainers, deliberately left out of scope: eval/rescore.py reads run-meta.json with the same unguarded json.loads() at lines 58, 102 and 279. Happy to follow up in a separate PR if that is wanted.

…summary

print_run_stats() parsed every run-meta.json with a bare json.loads(). A
single truncated or malformed file raised JSONDecodeError out of the
function, and because async_main() calls it at batch.py:753 *before*
write_summary_json() at :754, the exception took batch-summary.json down
with it -- one bad run destroyed the machine-readable results for every
other job in the batch.

The file gets truncated in the first place because write_run_meta() wrote
run-meta.json with a plain write_text(): a run killed mid-write (Ctrl-C,
OOM, host shutdown) leaves a half-written file behind.

Both halves are now fixed:

- New clawbench.utils.jsonio with write_json_atomic() (same-directory
  temp file, flush + fsync, os.replace) and read_json_or_none().
  os.replace is atomic on POSIX and Windows, so a crash mid-write leaves
  either the previous file or no file, never a truncated one.
- write_run_meta() and write_summary_json() write atomically.
- print_run_stats() reads via read_json_or_none(), also rejects valid
  JSON that is not an object (a bare list previously hit .get() and
  raised AttributeError), prints a warning naming the file, and falls
  back to the directory name for that one run.

Reporting now degrades to one warning line for the affected run instead
of losing the whole batch.

Fixes TIGER-AI-Lab#303.
@vaibhavdabas16
vaibhavdabas16 force-pushed the fix/batch-stats-atomic-run-meta branch from 758efda to 3b63507 Compare August 20, 2026 05:18
@Perry2004

Copy link
Copy Markdown
Collaborator

Thanks for the contribution. I'll take a look.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

One truncated run-meta.json aborts end-of-batch stats before batch-summary.json is written; metadata writes are not atomic

2 participants