[Superseded by #46/#159] Extend tabular/time-series scenario into generated-strategy benchmark

[dgenio]

Summary

Extend the existing foundation-model routing scenario into a benchmark for generated modelling strategies, not just model/workflow routing.

The new benchmark should compare candidate policies that decide whether to use a classical baseline, AutoML-style search, tabular foundation-model-style prediction, a generated constrained strategy, or a safe rejection/manual path.

Motivation

The recent research idea asks whether a model can one-shot produce a modelling strategy that beats strong human/AutoML/foundation-model baselines. agent-routing-eval-lab is a good place to prototype the routing/comparison side before anything becomes production library code.

Existing context:

• [Superseded by #46/#159] Foundation-model routing for tabular and time-series tasks #141 already introduces foundation-model routing for tabular/time-series tasks.
• This issue should deepen that into a generated-strategy benchmark.

Candidate strategies to compare

• baseline_router: conservative incumbent-like choice.
• automl_router: spends more compute to search.
• foundation_first_router: aggressively uses TabPFN/TabFM-style option where allowed.
• generated_strategy_router: picks a typed strategy specification.
• support_aware_router: rejects or asks for metadata when the task is under-specified.

Metrics

• task utility
• regret versus best observed strategy
• unsafe/leakage rate
• unnecessary compute rate
• rejection quality
• p95/p99 latency or cost proxy
• stability across task families

Acceptance criteria

• The scenario can run with synthetic fixtures.
• At least three strategy policies are compared.
• Report includes both aggregate score and per-task failure modes.
• The benchmark clearly distinguishes routing to a model from routing to a generated strategy.

Related

Builds on: #141