Roryd/deepscholar bench s2patch

[donovanr]

Description

Adds DeepScholar-Bench (arXiv 2508.20033) as a SandboxedExternalEval (deepscholar_bench). Given a paper's context, the system retrieves prior work and writes the related-work section, scored on organization, nugget_coverage, reference_coverage, and cite_p (geometric mean). Modeled on the tau2/asta external evals; clones the upstream repo and installs deps at setup, runs generation then eval in one sandbox.

Highlights:

• Local vLLM wiring via a generated LOTUS config (openai/ prefix through litellm); defaults to keyless recursive+arXiv retrieval for local models.
• sandbox_search_shim.py adds pluggable retrieval backends (-a search_backend=): arxiv (default), s2 (Semantic Scholar, keyed + backoff — robust at volume), tavily. The shim also lifts the upstream stage-LM max_tokens=512 cap that was truncating structured outputs and dropping ~30% of queries.
• Throttle/timeout knobs (search_steps, lm_timeout, stage_max_tokens) and strict-by-default generation (fails unless every query succeeds; allow_partial_generation to opt out).

Validated end to end on small runs (limit=2, limit=10): setup, generation, eval, and parsing all work; S2 backend produces well-formed citations (nonzero cite_p). Metrics look real, not artifacts — an early OLMo scores at the floor, as expected (no published system exceeds ~31% geomean).

Status / known issue

Blocked on full 63-query runs. 10-query runs are reliable (~20 min), but every attempt at the full set (runs 9-11) hangs ~40 min in: generation goes silent mid-LM-call and the sandbox watchdog kills the job, losing all completed queries. Raising max_model_len to 32768 fixed an earlier context-overflow cause, but the remaining stall is a time-correlated vLLM hang, not query-specific, and lm_timeout doesn't fire (litellm's streaming path appears to ignore it). This is a single-long-process reliability problem, not a correctness bug in the eval. Plan is to get the full number by batching (~6 jobs of ~12 queries, combine per-query CSVs) rather than chase the vLLM hang. Full run log in plans/006_deepscholar_bench.md.

Not wired into any suite yet; DEEPSCHOLAR_REF pinned to c95413b.

Type of Change

• New feature (non-breaking change that adds functionality)

Type of Change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update
• Refactoring (no functional changes)

Testing

• Unit tests pass locally (pytest tests/ --ignore=tests/integration/)
• Integration tests pass (if applicable)
• New tests added for new functionality

Checklist

• My code follows the project's style guidelines
• I have performed a self-review of my code
• I have added/updated documentation as needed
• My changes generate no new warnings
• Any dependent changes have been merged and published