Found during the pre-release bug-hunt. Follow-on to #57's expected-behavior line ("The max_documents limit should apply across all parsed lines") — the merged per-line loader enforces the cap within one file but not cumulatively across files.
In _load_documents' directory branch (openagent_eval/corpus/auditor.py ~188-202), the cumulative check if len(documents) >= self.max_documents: break sits only inside the non-jsonl else branch, so it never runs after a .jsonl file is processed; _load_jsonl_documents' own cap (line ~272) counts against that call's local list, which resets per file.
Repro: max_documents=2, directory with 2 jsonl files × 3 lines each:
loaded 4 documents (expected <= 2)
txt branch: 6 files, max=2 -> loaded 2 (cap enforced)
So the cap — a memory/safety bound — is silently bypassed and grows unbounded with more jsonl files, while the same directory of .txt files caps correctly. Fix is moving/duplicating the cumulative check to run after the jsonl branch too (and trimming the extended list to the cap).
Can PR immediately if useful.
Found during the pre-release bug-hunt. Follow-on to #57's expected-behavior line ("The max_documents limit should apply across all parsed lines") — the merged per-line loader enforces the cap within one file but not cumulatively across files.
In
_load_documents' directory branch (openagent_eval/corpus/auditor.py~188-202), the cumulative checkif len(documents) >= self.max_documents: breaksits only inside the non-jsonlelsebranch, so it never runs after a.jsonlfile is processed;_load_jsonl_documents' own cap (line ~272) counts against that call's local list, which resets per file.Repro:
max_documents=2, directory with 2 jsonl files × 3 lines each:So the cap — a memory/safety bound — is silently bypassed and grows unbounded with more jsonl files, while the same directory of .txt files caps correctly. Fix is moving/duplicating the cumulative check to run after the jsonl branch too (and trimming the extended list to the cap).
Can PR immediately if useful.