fix(batch): one truncated run-meta.json no longer discards the batch summary - #312
Open
vaibhavdabas16 wants to merge 1 commit into
Open
Conversation
…summary print_run_stats() parsed every run-meta.json with a bare json.loads(). A single truncated or malformed file raised JSONDecodeError out of the function, and because async_main() calls it at batch.py:753 *before* write_summary_json() at :754, the exception took batch-summary.json down with it -- one bad run destroyed the machine-readable results for every other job in the batch. The file gets truncated in the first place because write_run_meta() wrote run-meta.json with a plain write_text(): a run killed mid-write (Ctrl-C, OOM, host shutdown) leaves a half-written file behind. Both halves are now fixed: - New clawbench.utils.jsonio with write_json_atomic() (same-directory temp file, flush + fsync, os.replace) and read_json_or_none(). os.replace is atomic on POSIX and Windows, so a crash mid-write leaves either the previous file or no file, never a truncated one. - write_run_meta() and write_summary_json() write atomically. - print_run_stats() reads via read_json_or_none(), also rejects valid JSON that is not an object (a bare list previously hit .get() and raised AttributeError), prints a warning naming the file, and falls back to the directory name for that one run. Reporting now degrades to one warning line for the affected run instead of losing the whole batch. Fixes TIGER-AI-Lab#303.
vaibhavdabas16
force-pushed
the
fix/batch-stats-atomic-run-meta
branch
from
August 20, 2026 05:18
758efda to
3b63507
Compare
Collaborator
|
Thanks for the contribution. I'll take a look. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Fixes #303: a single truncated
run-meta.jsoncurrently destroys the machine-readable results for an entire batch, and the truncation itself is made impossible in the first place.The crash.
print_run_stats()parsed everyrun-meta.jsonwith a barejson.loads()(batch.py:450).async_main()calls it atbatch.py:753, one line beforewrite_summary_json()at:754— so one malformed file raisedJSONDecodeErrorout of the function and tookbatch-summary.jsondown with it. Every other job in the batch lost its recorded results, even though those runs succeeded.The truncation.
write_run_meta()(metadata.py:288) wrote with a plainwrite_text(). A run killed mid-write — Ctrl-C, OOM, host shutdown — leaves a half-written file on disk, which is precisely what the reader above then chokes on.Both halves are addressed:
clawbench.utils.jsoniowithwrite_json_atomic()(same-directory temp file, flush +fsync, thenos.replace) andread_json_or_none().os.replacereplaces atomically on POSIX and Windows, so an interrupted write leaves either the previous file or no file — never a truncated one.write_run_meta()andwrite_summary_json()now write atomically.print_run_stats()reads viaread_json_or_none(). It also rejects valid JSON that isn't an object — a bare list previously reached.get()and raisedAttributeError— prints a warning naming the offending file, and falls back to the directory name for that one run.Reporting now degrades to a single warning line for the affected run instead of losing the whole batch.
Corpus
Host-side runner/reporting change; no task data involved.
Test plan
tests/test_batch_stats_resilience.py(11 tests): atomic round-trip and parent-dir creation, no leftover temp files, previous file survives a failed serialization,read_json_or_none()against truncated / empty / garbage / undecodable input, and twoprint_run_stats()regression tests covering a truncatedrun-meta.jsonand a non-object one.batch.pyandmetadata.pytomainand re-ran, which fails withJSONDecodeErrorandAttributeErrorrespectively; both pass with the fix applied.203 passed, 3 skipped. The one unrelated failure,test_host_tasks.py::test_checked_task_json_files_parse_and_validate[v1-lite], reproduces identically on a cleanmaincheckout on this machine — thev1-litetask files are git symlinks (mode120000) that Windows checks out as plain text.ruff checkandruff format --checkclean on all changed files. Removing the lastjson.uses frombatch.pyleftimport jsonunused, so it is dropped (F401).Related issues
Fixes #303.
One note for maintainers, deliberately left out of scope:
eval/rescore.pyreadsrun-meta.jsonwith the same unguardedjson.loads()at lines 58, 102 and 279. Happy to follow up in a separate PR if that is wanted.