Explore ML experiment track: ScienceAgentBench as generalization proof

[Darkroom4364]

Motivation

Noeris's core architecture, persistent cross-run memory, learned cost model, adaptive strategy routing, parameterized search, was designed for GPU kernel optimization but is domain-agnostic by construction. The thesis doc already states the company vision is "autonomous empirical discovery" with kernels as the wedge.

The science/ML experiment benchmark space is wide open and structurally weak in exactly the ways Noeris is strong.

The Opportunity

ScienceAgentBench (OSU NLP, ICLR 2025): 102 data-driven science tasks across bioinformatics, chemistry, GIS, and psychology.

• SOTA: 33.3% (o3, $11.69/task). Stagnant, no submission has broken 35%.
• Best open agent: OpenHands CodeAct at 32.4%. It's a linear ReAct agent with zero cross-task memory.
• Evaluation is deterministic (output matches gold reference) — no seed variance / noisy eval problem.
• Lite subset: 32 tasks, ~$15-50 per dev run. CPU-only compute. Fast iteration.

Failure analysis from the paper:

• 30% domain knowledge gaps (wrong algorithm / method choice)
• 25% multi-step planning failures (lose coherence at step 7-10)
• 20% API/library misuse (wrong params, deprecated calls)
• 15% output format mismatch
• 10% dataset misunderstanding

These are memory and planning failures, not model failures. Cross-task persistent memory directly addresses the top 3 categories.

Supporting evidence:

• AgentRxiv (ETH Zurich): shared memory between agent runs gave 13.7% improvement on MATH-500. Performance broke through a plateau.
• FML-bench: exploration breadth > exploitation depth. Agents with diverse strategies outperform those that repeatedly refine.
• MLE-STAR (Google): component-level refinement (don't rewrite everything, target specific pipeline stages) jumped from 25.8% to 63.6% on MLE-bench.

Architecture Mapping

Noeris (kernels)	ML experiment equivalent	Reuse
OperatorRegistry + TritonOperatorSpec	TaskRegistry + ScienceTaskSpec (4 disciplines, strategy space per discipline)	~70%
ConfigDatabase keyed by (operator, shape, hw)	ExperimentDatabase keyed by (discipline, task_type, approach)	~85%
CostModel (GBR on config features)	Skip for v0 (experiments are $0.50-2 each). Add when DB has enough data	Deferred
AdaptiveRouter (bandit over selectors)	Bandit over strategies: direct_solve vs decompose_subtasks vs retrieve_and_adapt	~90%
BanditSelector (Thompson sampling)	Same, over approach selection	~95%
ResearchPipeline (7 components)	Same shape, new component implementations	~90%
modal_runner.py	Sandboxed Python runner (E2B or local Docker), CPU-only	~30% (rewrite)
llm.py (proposer with insights)	Proposer with cross-task insights + domain library priors	~50%

Estimated reuse: 60-70% of existing architecture. The genuinely new engineering:

1. Python sandbox runner (replacing Modal GPU runner)
2. Domain-specific strategy specs (library priors, known pitfalls, subtask templates)
3. Cross-task memory extraction ("scanpy normalize_total needs target_sum=1e4")

What the Strategy Space Looks Like

Not numerical param grids like kernels — categorical approach selection + accumulated domain knowledge:

@dataclass
class ScienceStrategySpec:
    name: str  # "bioinformatics"
    solution_strategies: list[str]  # direct_gen, decompose, template_fill, retrieve_adapt
    library_priors: dict[str, list[str]]  # {"single_cell_rna": ["scanpy", "anndata"]}
    known_pitfalls: list[str]  # accumulated from past task failures
    subtask_templates: dict[str, list[str]]  # {"figure_gen": ["load", "process", "compute", "plot"]}

Learned from execution, not hand-curated (unlike KernelSkill's static skill library).

Scope for v0

Target: Beat 33.3% SR on ScienceAgentBench-Lite (32 tasks).

Build:

1. TaskRegistry with 4 discipline buckets
2. ExperimentDatabase accumulating across the 32 tasks
3. AdaptiveRouter with 3 arms (direct_solve, decompose, retrieve_and_adapt)
4. Sandboxed Python runner (Docker or E2B)
5. Cross-task memory extraction after each task

Skip for v0: Cost model (experiments too cheap), web retrieval, MLE-bench integration.

Success criterion: >35% SR on lite subset. Even 36-38% would be publishable given 18 months of stagnation at 33%.

Estimated effort: 2-3 weeks on top of existing Noeris infrastructure.

Why This Matters for the Paper

The current paper's weakness: "we proved this works for kernels but haven't shown it generalizes." Adding an ML experiment track with the same architecture beating SOTA on a different benchmark turns the paper from "a kernel optimization system" into "a general architecture for autonomous empirical search."

Follow-up Targets

• MLGym (Meta FAIR): 13 open-ended ML research tasks. All frontier models plateau at Level 1 (hyperparameter tuning only). Breaking the L1→L2 barrier (component swapping, novel combinations) using Noeris's hypothesis→experiment→analysis cycle would be a significant result. Harder, more prestigious.
• MLE-bench: Active leaderboard but expensive ($2.5-5K per Lite eval) and crowded. Model capability is the dominant factor. Lower priority.

Key References

• ScienceAgentBench: arXiv 2410.05080, HAL leaderboard
• MLGym: arXiv 2502.14499, GitHub
• AgentRxiv (cross-run memory): arXiv 2503.18102
• FML-bench (breadth > depth): arXiv 2510.10472
• 12-Factor Agents: humanlayer.dev/12-factor-agents
• AIDE: GitHub

[Darkroom4364]

Local deterministic ScienceAgentBench scaffold started in issue #117.

Implemented path:

• src/research_engine/science_agent_bench.py
• tests/test_science_agent_bench.py
• docs/research/scienceagentbench-local-scaffold.md

Next concrete benchmark-integration step: add an adapter that converts one ScienceAgentBench-Lite task fixture into ScienceTaskSpec, runs it in an offline sandbox, and compares the produced artifact to the benchmark gold output. No ScienceAgentBench score should be reported until that adapter runs real benchmark tasks.