Bake SciCode sandbox image once; pre-build from launch host

[finbarrtimbers]

Stacks on #153 and #160 — both must merge first.

Previously, every SciCode sub-step launch ran uv sync inside the sandbox to install numpy / scipy / sympy / h5py from a mounted lockfile, blocking the GPU on hundreds of redundant resolves. This PR bakes the deps into the sandbox image once and pre-builds it on the launch host before submitting the Beaker job, so the GPU node only has to pull a ready image.

Mechanism

• SciCodeExternalEval.prebuild_sandbox_specs() declares its lockfile as dockerfile_extra lines on the swerex base image. The dockerfile_extra plumbing is already hashed into get_swerex_image's deterministic tag (src/olmo_eval/harness/sandbox/image.py:55-57), so dep changes invalidate the cache automatically.
• The Beaker launcher runs _prebuild_external_eval_sandbox_images before submitting (src/olmo_eval/cli/beaker/launch.py), calling get_swerex_image(..., require_registry=True) on the user's machine. The launch host always pushes to BEAKER_INFRA_ENV_VARS["SWEREX_REGISTRY"] — the same registry the Beaker job pulls from — so users don't have to set SWEREX_REGISTRY in their shell.
• ExternalEval.prebuild_sandbox_specs() defaults to returning (), so non-SciCode external evals are no-op pass-throughs.

Measurements

• Local: 25.3s first sandbox (image build), 1.6s on cache hit.
• Previously: 5–30s per sandbox at runtime, multiplied across max_concurrency × n_problems.

Circular-import dependency

The new top-level from olmo_eval.harness.sandbox import SandboxManager in scicode/eval.py triggers a cycle through runners.io.storage → cli.utils → cli/__init__ → harness.presets. The fix lives in #160 (move shared console to olmo_eval.common.console) — split out for independent review. Once #160 merges, this branch should be rebased to drop its duplicate of that commit.

Verification

• uv run pytest tests/evals/external/benchmarks/test_scicode_external.py tests/cli — 54/54 pass, including a new TestPrebuildSandboxSpecs.
• make lint clean.
• Real Beaker launch of Phi-4-mini-instruct on SciCode (Add SciCode #153 baseline) showed sandbox starts within 1–2 s instead of 5–30 s.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0fa3685c6b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Install SciCode deps into the active sandbox venv

This uv sync --active command now runs during image build, but in get_swerex_image the injected dockerfile_extra lines execute before VIRTUAL_ENV/PATH are set to /root/venv (src/olmo_eval/harness/sandbox/image.py). In that context, uv creates a project-local .venv under /opt/scicode-deps instead of installing into the runtime venv used by sandbox execution, so SciCode verification can hit missing numpy/scipy/sympy/h5py imports after the runtime uv sync step was removed.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Avoid making sandbox pre-bake require registry push access

Calling get_swerex_image(..., require_registry=True) makes pre-bake a hard prerequisite for launch: if the launch host lacks push auth to SWEREX_REGISTRY, this raises and aborts beaker launch before any job is submitted. That is a regression from prior behavior where the Beaker job could still build and run the derived image locally on the GPU node (with require_registry=False) even when registry push failed.

Useful? React with 👍 / 👎.