[Superseded by #46/#159] Add tabular data-science agent routing benchmark scenario

[dgenio]

Context

Tabular ML agent performance depends on routing attention to the right kind of work:

• inspect schema;
• profile data;
• choose a baseline;
• check leakage;
• generate feature hypotheses;
• call domain tools;
• stop and escalate when evidence is insufficient.

The interesting routing question is not only "which model should run?". It is also "which data-science action should happen next?"

Related existing issues:

• [Superseded by #46/#159] Foundation-model routing for tabular and time-series tasks #141 — foundation-model routing for tabular and time-series tasks
• [Superseded by #46/#159] Extend tabular/time-series scenario into generated-strategy benchmark #142 — generated-strategy benchmark
• [Superseded by #46/#159] Add synthetic tabular task families with known failure modes #143 — synthetic tabular task families with known failure modes

This issue adds a more agentic routing scenario focused on next-action selection inside a tabular data-science workflow.

Goal

Create a benchmark scenario where candidate policies choose the next action for a tabular data-science agent.

Candidate actions

The router/policy may choose between actions such as:

• profile_schema
• inspect_missingness
• detect_leakage_risk
• choose_baseline_model
• generate_feature_hypothesis
• run_feature_transform
• evaluate_baseline
• compare_experiment_runs
• request_domain_context
• escalate_to_human
• stop_and_report

Scenario families

Add small synthetic or fixture-backed cases where the best next action differs:

1. Easy IID baseline

   • Correct behaviour: run a simple baseline and stop if sufficient.

2. Leakage trap

   • Correct behaviour: detect leakage before celebrating high metrics.

3. Grouped entity split

   • Correct behaviour: use grouped split instead of random split.

4. Temporal leakage

   • Correct behaviour: require time-aware validation.

5. Domain feature needed

   • Correct behaviour: generate/test a domain-aware feature hypothesis before model sweeping.

6. Insufficient context

   • Correct behaviour: ask for domain information or escalate.

Metrics

Evaluate policies on:

• correct next-action rate;
• unsafe-action rate;
• leakage-missed rate;
• unnecessary-model-sweep rate;
• premature-stop rate;
• escalation precision/recall;
• cost proxy.

Acceptance criteria

• New scenario exists for tabular DS next-action routing.
• At least three scenario families are implemented or specified as fixtures.
• Evaluation report includes safety/quality metrics beyond accuracy.
• At least one bad baseline policy demonstrates the failure mode of model-sweeping too early.
• README/docs explain how this differs from generic AutoML benchmarking.

Non-goals

• Run expensive Kaggle-scale benchmarks.
• Train large models.
• Claim that the benchmark captures all data-science judgement.