diff --git a/TASK_DETAILS.md b/TASK_DETAILS.md
index 7475c4cc..66574c78 100644
--- a/TASK_DETAILS.md
+++ b/TASK_DETAILS.md
@@ -351,5 +351,10 @@ We welcome new engineering problem ideas — even without complete verification
DiffSimThermalControl |
Process optimization in additive manufacturing via differentiable simulation |
+
+ | VehicleRouting |
+ CVRP |
+ Capacitated VRP: route a homogeneous fleet from a single depot, every customer served once within capacity, minimize total distance |
+
diff --git a/TASK_DETAILS_zh-CN.md b/TASK_DETAILS_zh-CN.md
index e2a070a2..d4974555 100644
--- a/TASK_DETAILS_zh-CN.md
+++ b/TASK_DETAILS_zh-CN.md
@@ -351,5 +351,10 @@ Frontier-Eng 目前已覆盖以下领域的任务。每个任务均配有可运
DiffSimThermalControl |
基于可微仿真的增材制造工艺优化 |
+
+ | VehicleRouting |
+ CVRP |
+ 容量约束车辆路径：单一仓库、同型车队、每客户服务一次且不超容量，最小化总距离 |
+
diff --git a/benchmarks/VehicleRouting/CVRP/.gitignore b/benchmarks/VehicleRouting/CVRP/.gitignore
new file mode 100644
index 00000000..7a60b85e
--- /dev/null
+++ b/benchmarks/VehicleRouting/CVRP/.gitignore
@@ -0,0 +1,2 @@
+__pycache__/
+*.pyc
diff --git a/benchmarks/VehicleRouting/CVRP/README.md b/benchmarks/VehicleRouting/CVRP/README.md
new file mode 100644
index 00000000..0807501a
--- /dev/null
+++ b/benchmarks/VehicleRouting/CVRP/README.md
@@ -0,0 +1,244 @@
+# CVRP (Capacitated Vehicle Routing Problem) Benchmark
+
+Standard CVRP: a fleet of identical vehicles serves all customers from a
+single depot, each route within vehicle capacity, minimizing total distance.
+
+## Structure
+
+```
+CVRP/
+├── Task.md # Task description (rules, I/O contract, scoring)
+├── baseline/
+│ ├── solver.py # Candidate solver (random-order cheapest insertion) + EVOLVE-BLOCK
+│ └── result_log.txt # Baseline evaluation log
+├── verification/
+│ ├── evaluator.py # Runner + validator + scorer + candidate integrity checks (stdlib only)
+│ ├── validator.py # Candidate integrity validator (static checks + determinism probe)
+│ ├── ref_solver.py # Reference solver: deterministic GRASP multi-start + 2-opt + relocate/swap + 2-opt* + LNS
+│ ├── generate_instances.py# Deterministic instance generator (seed 42, byte-stable; public + held-out)
+│ ├── test_evaluator.py # Unit tests: evaluator (stdlib unittest)
+│ ├── test_validator.py # Unit tests: validator
+│ ├── test_ref_solver.py # Unit tests: reference solver
+│ ├── multiseed_stat.py # Multi-seed baseline statistics script
+│ └── requirements.txt
+├── data/
+│ ├── instances/ # 12 public TSPLIB-style .vrp instances (VRP-*)
+│ ├── instances_heldout/ # 12 held-out instances (VHO-*), scored at evaluation time
+│ └── reference.json # Precomputed reference distance per instance (24)
+└── frontier_eval/ # Unified-task metadata
+```
+
+## Requirements
+
+Python 3 standard library only. No third-party dependencies.
+
+## How to run
+
+```bash
+# Evaluate a candidate solver (inside the CVRP directory)
+python verification/evaluator.py baseline/solver.py
+
+# Unified-task adapter check (repo root)
+python -m frontier_eval task=unified task.benchmark=VehicleRouting/CVRP algorithm.iterations=0
+```
+
+### Docker
+
+The Dockerfile provides the evaluation environment (Python stdlib) for the
+unified runtime's `isolation_mode=docker`. Build it and point the unified
+runtime at it:
+
+```bash
+# Build the image (inside the CVRP directory)
+docker build -t cvrp-benchmark -f verification/docker/Dockerfile .
+
+# Use it from the repo root (see "Experiments" for the full command)
+python -m frontier_eval task=unified task.benchmark=VehicleRouting/CVRP algorithm.iterations=0 task.runtime.isolation_mode=docker task.runtime.docker_image=cvrp-benchmark
+```
+
+> Docker isolation is validated on Linux / WSL. `frontier_eval/eval_command.txt`
+> injects the host-benchmark path via the `{benchmark_source}` placeholder, so
+> scoring works without framework changes; if the container user cannot write
+> the evaluation sandbox, set `task.runtime.docker_user=:`
+> (e.g. `1000:1000`). On Windows hosts the unified docker path is blocked by a
+> framework path bug (`Path.resolve()` rewrites container paths to drive
+> paths) — run docker mode under WSL instead.
+
+The image intentionally contains no benchmark files (no `reference.json`, no
+reference solver); the unified runtime mounts the sandbox into the container.
+
+## Unified-task integration
+
+- Benchmark id: `VehicleRouting/CVRP`
+- The evaluator uses only the Python standard library, so no runtime overrides
+ (`python_path`, conda env, or Docker image) are required; `eval_command.txt`
+ uses plain `{python}`.
+- Windows only: local validation must point `task.runtime.shell` at Git Bash
+ (e.g. `task.runtime.shell=C:/Program Files/Git/bin/bash.exe`), because the
+ default `bash` resolves to the WSL shim without `python` (returncode 127).
+ Linux needs no override.
+
+## Scoring
+
+`score = min(100, 100 * reference_distance / candidate_distance)` averaged over
+24 instances (12 public + 12 held-out). The reference distances are
+precomputed by the deterministic `verification/ref_solver.py` and are
+near-optimal (cross-checked against the archived best agent solver and an
+OR-Tools GLS solve). Invalid solutions (missing/duplicate customers, capacity
+violations, crashes, timeouts) score 0 and mark the run invalid. An optional
+`CVRP_EVAL_SCORE_SCALE` knob (default 1.0) tightens the 100-point bar:
+`score = min(100, scale * 100 * ref / cand)`.
+
+## Reference scores (measured on this machine)
+
+The current evaluation set is 24 instances (12 public + 12 held-out). Most
+agent scores below were measured on the earlier 12-public-instance set (before
+held-out instances were added) and are kept for cross-framework comparison;
+fresh runs on the full 24-instance set (ShinkaEvolve 98.65, openevolve 98.00,
+AB-MCTS 99.26) show the learned solvers generalize to unseen instances. See
+"Experiments" for run records and multi-run statistics.
+
+| Solver | combined_score |
+|--------|----------------|
+| baseline (random-order cheapest insertion), 24 instances | 54.69 |
+| reference (deterministic GRASP + LNS, scoring baseline) | 100 (near-optimal) |
+| agent (openevolve, 5 iterations, best, 12-instance set) | 96.38 |
+| agent (openevolve, 5 iterations, best, **24-instance set**) | **98.00** |
+| agent (ShinkaEvolve, 5 generations, best, 12-instance set) | 99.31 |
+| agent (ShinkaEvolve, 5 generations, best, **24-instance set**) | **98.65** |
+| agent (AB-MCTS, 5 candidates, best, 12-instance set) | 98.70 |
+| agent (AB-MCTS, 5 candidates, best, **24-instance set**) | **99.26** |
+
+Baseline score distribution over the 24 instances
+(`python verification/evaluator.py baseline/solver.py`): mean **54.69**,
+std 7.32, min 44.03 (`VHO-51-7`), max 71.55 (`VRP-21-3`). The solver is
+deterministic (seeded RNG), so repeated runs are byte-identical; the spread
+above is across instances, not across seeds.
+
+**Multi-seed statistics** (reviewer-requested "multi-run statistics"):
+`verification/multiseed_stat.py` derives fresh evaluation instance sets from
+multiple seeds and computes each set's reference distances on the fly
+(`python verification/multiseed_stat.py --seeds 111 222 333`):
+
+| Seed | combined_score |
+|------|----------------|
+| 111 | 57.24 |
+| 222 | 59.86 |
+| 333 | 56.09 |
+| **mean ± std** | **57.73 ± 1.58** (min 56.09, max 59.86) |
+
+**Agent multi-run statistics** (3 runs per framework on the 24-instance set,
+`deepseek-v4-flash`; full run IDs in "Experiments"):
+
+| Framework | runs (combined_score) | mean | std | min | max |
+|-----------|-----------------------|------|-----|-----|-----|
+| openevolve (5 iterations) | 98.00, 97.96, 97.47 | **97.81** | 0.24 | 97.47 | 98.00 |
+| ShinkaEvolve (5 generations) | 98.13, 54.69\*, 98.65 | **83.82** | 20.60 | 54.69 | 98.65 |
+| AB-MCTS (5 candidates) | 98.49, 96.78, 99.26 | **98.18** | 1.04 | 96.78 | 99.26 |
+
+\* one ShinkaEvolve run produced invalid programs in every generation (each
+scored 0), so its best stayed at the baseline 54.69 — a genuine failure mode,
+not an environment issue.
+
+Agent scores are "best found" over stochastic evolution runs; multiple runs
+are listed in the "Experiments" table where available (e.g. openevolve 96.38
+and 95.65).
+
+## Evaluation integrity
+
+- **Held-out instances**: 12 `VHO-*` instances in `data/instances_heldout/`
+ double the evaluation set to 24 and are scored alongside the public ones.
+ Their `.vrp` files are **not** copied into the evaluation sandbox: the
+ evaluator reads the paths from the host
+ (`FRONTIER_EVAL_UNIFIED_SOURCE_BENCHMARK_DIR`) and hands each to the
+ candidate only while that instance is being scored. The files are public in
+ the repo, and a running candidate receives the path and can read the file
+ (see the threat model below); `Task.md` and `constraints.txt` tell the agent
+ such held-out instances exist and will be scored, but their data is not
+ available at code-generation time, so an LLM cannot hardcode routes by
+ instance name. Per-instance name-keyed hardcoding is additionally rejected
+ statically by `validator.py`, and `CVRP_EVAL_GENERATE_SEED` adds fresh
+ instances at evaluation time so the scored set is not predictable.
+- **Runtime-generated instances**: set `CVRP_EVAL_GENERATE_SEED` (and
+ optionally `CVRP_EVAL_GENERATE_COUNT`, default 6) to additionally generate
+ fresh instances at evaluation time from that seed. Each generated instance
+ is scored against a reference computed on the fly by the reference solver,
+ so a candidate cannot memorize the evaluation set even if it has seen every
+ public instance file. Same seed ⇒ same instances ⇒ reproducible. Works in
+ the direct evaluator and the unified runtime in process mode. In docker
+ isolation mode (Linux / WSL) scoring works via the `{benchmark_source}` env
+ injection in `eval_command.txt`, but runtime-generated instances are not
+ available there because the unified runtime does not forward the seed env
+ var into the container (framework-level limitation).
+- **Sandbox**: `copy_files.txt` copies only `baseline/`, `data/instances/` and
+ `frontier_eval/` into the evaluation sandbox (held-out instances and
+ `reference.json` are read from the host, never copied).
+ `frontier_eval/evaluator.py` is self-contained (parsing, validation,
+ scoring and integrity checks are embedded), so no `verification/` files —
+ including the reference solver — are copied. `reference.json` is never
+ copied; the evaluator reads it from the host benchmark dir
+ (`FRONTIER_EVAL_UNIFIED_SOURCE_BENCHMARK_DIR`), and the candidate subprocess
+ runs with **all `FRONTIER_*` variables removed** (plus reference/generation
+ settings), so it cannot learn the host repo path — this closes the
+ `FRONTIER_ENGINEERING_ROOT` side channel that would otherwise let a candidate
+ find and import `verification/ref_solver.py` on the host.
+- **Preflight checks** (`verification/validator.py`): the evaluator statically
+ rejects candidates that modify code outside the EVOLVE-BLOCK, reference
+ `verification` / `ref_solver` / `reference.json`, contain absolute paths, or
+ hardcode per-instance routes; a determinism probe runs the candidate twice
+ on small / medium / large probe instances and invalidates non-deterministic
+ solvers.
+- **Threat model**: all instance files (public and held-out) are visible in
+ the repo and in the evaluation sandbox — a human who can read the repo could
+ always hand-craft a solver, which no benchmark can prevent. The protections
+ are deterrence-level: the constraints forbid reading the reference and
+ hardcoding by instance name, `validator.py` statically rejects such
+ attempts, and `CVRP_EVAL_GENERATE_SEED` makes the scored instance set
+ unpredictable at evaluation time. The unified runtime (process and docker
+ isolation) exposes the host repo to the candidate process (framework-level
+ behavior shared by all tasks). Scoring only measures solution quality, never
+ where the code came from.
+
+## Experiments
+
+All agent runs use the `deepseek-v4-flash` model. Scores are "best found";
+LLM evolution is stochastic, so multiple runs are listed where available.
+
+| Run | Framework | Score | Set |
+|-----|-----------|-------|-----|
+| baseline (random-order cheapest insertion) | — | 54.69 | 24 instances (12 public + 12 held-out) |
+| reference (GRASP + LNS, scoring baseline) | — | 100 | 24 instances |
+| `runs/.../openevolve/deepseek-v4-flash/20260807_170244` | openevolve, 5 iterations | 96.38 | 12 public |
+| `runs/.../openevolve/deepseek-v4-flash/20260807_195321` | openevolve, 5 iterations | 95.65 | 12 public |
+| `runs/.../openevolve/deepseek-v4-flash/20260811_215703` | openevolve, 5 iterations | 98.00 | 24 instances (12 public + 12 held-out) |
+| `runs/.../openevolve/deepseek-v4-flash/20260812_172944` | openevolve, 5 iterations | 97.96 | 24 instances (12 public + 12 held-out) |
+| `runs/.../openevolve/deepseek-v4-flash/20260812_174122` | openevolve, 5 iterations | 97.47 | 24 instances (12 public + 12 held-out) |
+| `runs/.../shinkaevolve/deepseek-v4-flash/20260807_195503` | ShinkaEvolve, 5 generations | 99.31 | 12 public |
+| `runs/.../shinkaevolve/deepseek-v4-flash/20260811_192446` | ShinkaEvolve, 5 generations | 98.13 | 24 instances (12 public + 12 held-out) |
+| `runs/.../shinkaevolve/deepseek-v4-flash/20260812_175208` | ShinkaEvolve, 5 generations | 54.69\* | 24 instances (12 public + 12 held-out) |
+| `runs/.../shinkaevolve/deepseek-v4-flash/20260812_175816` | ShinkaEvolve, 5 generations | 98.65 | 24 instances (12 public + 12 held-out) |
+| `runs/.../abmcts/deepseek-v4-flash/20260807_190515` | AB-MCTS, 5 candidates | 98.70 | 12 public |
+| `runs/.../abmcts/deepseek-v4-flash/20260812_102741` | AB-MCTS, 5 candidates | 98.49 | 24 instances (12 public + 12 held-out) |
+| `runs/.../abmcts/deepseek-v4-flash/20260812_181308` | AB-MCTS, 5 candidates | 96.78 | 24 instances (12 public + 12 held-out) |
+| `runs/.../abmcts/deepseek-v4-flash/20260812_182222` | AB-MCTS, 5 candidates | 99.26 | 24 instances (12 public + 12 held-out) |
+
+Baseline reproduction:
+
+```bash
+# inside the CVRP directory
+python verification/evaluator.py baseline/solver.py # -> 54.69, valid 1.0 (24 instances)
+python verification/test_evaluator.py # -> 23 unit tests pass
+python verification/test_validator.py # -> 16 unit tests pass
+python verification/test_ref_solver.py # -> 5 unit tests pass
+python verification/test_frontier_eval_evaluator.py # -> 6 unit tests pass (sandbox copy)
+```
+
+Unified adapter checks (repo root):
+
+```bash
+# process mode
+python -m frontier_eval task=unified task.benchmark=VehicleRouting/CVRP algorithm.iterations=0
+
+# docker isolation (build first: docker build -t cvrp-benchmark -f verification/docker/Dockerfile .)
+python -m frontier_eval task=unified task.benchmark=VehicleRouting/CVRP algorithm.iterations=0 task.runtime.isolation_mode=docker task.runtime.docker_image=cvrp-benchmark
+```
diff --git a/benchmarks/VehicleRouting/CVRP/README_zh-CN.md b/benchmarks/VehicleRouting/CVRP/README_zh-CN.md
new file mode 100644
index 00000000..58cf53d3
--- /dev/null
+++ b/benchmarks/VehicleRouting/CVRP/README_zh-CN.md
@@ -0,0 +1,165 @@
+# CVRP（容量约束车辆路径问题）Benchmark
+
+标准 CVRP：一支同型车队从单一仓库出发服务所有客户，每条路线不超过车辆容量，目标是最小化总行驶距离。
+
+## 目录结构
+
+```
+CVRP/
+├── Task.md # 任务说明（规则、输入输出契约、评分）
+├── baseline/
+│ ├── solver.py # 候选求解器（随机顺序最近插入）+ EVOLVE-BLOCK
+│ └── result_log.txt # baseline 评测日志
+├── verification/
+│ ├── evaluator.py # 运行器 + 验证器 + 评分器 + 候选完整性检查（仅标准库）
+│ ├── validator.py # 候选完整性验证器（静态检查 + 确定性探针）
+│ ├── ref_solver.py # 参考求解器：确定性 GRASP 多起点 + 2-opt + relocate/swap + 2-opt* + LNS
+│ ├── generate_instances.py# 确定性实例生成器（seed 42，逐字节稳定；公开 + held-out）
+│ ├── test_evaluator.py # 单元测试：评测器（stdlib unittest）
+│ ├── test_validator.py # 单元测试：验证器
+│ ├── test_ref_solver.py # 单元测试：参考求解器
+│ ├── multiseed_stat.py # baseline 多 seed 统计脚本
+│ ├── requirements.txt
+│ └── docker/
+│ └── Dockerfile # 容器化评测环境（Docker 优先）
+├── data/
+│ ├── instances/ # 12 个公开 TSPLIB 风格 .vrp 实例（VRP-*）
+│ ├── instances_heldout/ # 12 个 held-out 实例（VHO-*），评测时评分
+│ └── reference.json # 每个实例的参考距离（24 个，评分基准）
+└── frontier_eval/ # Unified-task 元数据
+```
+
+## 依赖
+
+仅需 Python 标准库，无第三方依赖。
+
+## 如何运行
+
+```bash
+# 评测一个候选求解器（在 CVRP 目录下）
+python verification/evaluator.py baseline/solver.py
+
+# 框架适配验证（仓库根目录）
+python -m frontier_eval task=unified task.benchmark=VehicleRouting/CVRP algorithm.iterations=0
+```
+
+### Docker
+
+Dockerfile 为 unified 运行时的 `isolation_mode=docker` 提供评测环境（Python 标准库）。构建镜像并让 unified 运行时指向它：
+
+```bash
+# 构建镜像（在 CVRP 目录下）
+docker build -t cvrp-benchmark -f verification/docker/Dockerfile .
+
+# 从仓库根目录使用（完整命令见"实验记录"）
+python -m frontier_eval task=unified task.benchmark=VehicleRouting/CVRP algorithm.iterations=0 task.runtime.isolation_mode=docker task.runtime.docker_image=cvrp-benchmark
+```
+
+> Docker 隔离在 **Linux / WSL** 下已验证可用。`frontier_eval/eval_command.txt` 通过 `{benchmark_source}` 占位符把宿主 benchmark 路径注入评测命令，评分无需改框架即可工作；若容器用户无法写入评测沙箱，设置 `task.runtime.docker_user=<宿主 uid>:<宿主 gid>`（如 `1000:1000`）。Windows 宿主的 unified docker 路径被框架的路径 bug 卡住（`Path.resolve()` 会把容器路径改写为盘符路径）——请在 WSL 下运行 docker 模式。
+
+镜像内**刻意不包含任何 benchmark 文件**（没有 `reference.json`、没有参考求解器）；unified 运行时会把沙箱挂载进容器。
+
+## 评分
+
+`score = min(100, 100 × reference_distance / candidate_distance)`，跨 **24 个实例**（12 公开 + 12 held-out）取平均。
+参考距离由确定性的 `verification/ref_solver.py` 预计算，为近最优（与存档最优 agent 解、OR-Tools GLS 交叉验证一致）。
+非法解（客户缺失/重复、超容量、崩溃、超时）得 0 分，并使整个运行判无效（valid=0）。
+可选评分旋钮 `CVRP_EVAL_SCORE_SCALE`（默认 1.0）可收紧 100 分线：`score = min(100, scale × 100 × ref / cand)`。
+
+## 参考分数（本机实测）
+
+当前评测集为 24 个实例（12 公开 + 12 held-out）。下面多数 agent 分数是在较早的 12 公开实例集上测得的（held-out 实例为后加），保留用于跨框架对比；在完整 24 实例集上的新运行（ShinkaEvolve 98.65、openevolve 98.00、AB-MCTS 99.26）证明学到的求解器能泛化到未见过的实例。运行记录与多运行统计见"实验记录"。
+
+| 求解器 | combined_score |
+|--------|----------------|
+| baseline（随机顺序最近插入），24 实例 | **54.69** |
+| reference（确定性 GRASP + LNS，评分基准） | 100（近最优） |
+| agent（openevolve，5 轮，best，12 实例集） | 96.38 |
+| agent（openevolve，5 轮，best，**24 实例集**） | **98.00** |
+| agent（ShinkaEvolve，5 代，best，12 实例集） | 99.31 |
+| agent（ShinkaEvolve，5 代，best，**24 实例集**） | **98.65** |
+| agent（AB-MCTS，5 候选，best，12 实例集） | 98.70 |
+| agent（AB-MCTS，5 候选，best，**24 实例集**） | **99.26** |
+
+baseline 在 24 个实例上的分数分布（`python verification/evaluator.py baseline/solver.py`）：mean **54.69**、std 7.32、min 44.03（`VHO-51-7`）、max 71.55（`VRP-21-3`）。求解器是确定性的（固定种子 RNG），重复运行逐字节一致；上述离散度是跨实例的，而非跨 seed 的。
+
+**多 seed 统计**（评审要求的 "multi-run statistics"）：`verification/multiseed_stat.py` 从多个 seed 派生全新评测实例集并现场计算各集参考距离（`python verification/multiseed_stat.py --seeds 111 222 333`）：
+
+| Seed | combined_score |
+|------|----------------|
+| 111 | 57.24 |
+| 222 | 59.86 |
+| 333 | 56.09 |
+| **mean ± std** | **57.73 ± 1.58**（min 56.09，max 59.86） |
+
+**Agent 多运行统计**（每框架 3 次运行，24 实例集，`deepseek-v4-flash`；完整运行 ID 见"实验记录"）：
+
+| 框架 | 各次 combined_score | mean | std | min | max |
+|------|---------------------|------|-----|-----|-----|
+| openevolve（5 轮） | 98.00, 97.96, 97.47 | **97.81** | 0.24 | 97.47 | 98.00 |
+| ShinkaEvolve（5 代） | 98.13, 54.69\*, 98.65 | **83.82** | 20.60 | 54.69 | 98.65 |
+| AB-MCTS（5 候选） | 98.49, 96.78, 99.26 | **98.18** | 1.04 | 96.78 | 99.26 |
+
+\* 有一次 ShinkaEvolve 运行在每一代都生成了无效程序（各代计 0 分），其 best 停在 baseline 54.69——这是真实的失败模式，不是环境问题。
+
+agent 分数为随机进化运行的 "best found"；有多次运行的一并列出（如 openevolve 96.38 与 95.65，见"实验记录"）。
+
+## 评测完整性
+
+- **Held-out 实例**：12 个 `VHO-*` 实例位于 `data/instances_heldout/`，把评测集扩充到 24 个并与公开实例一起评分。它们的 `.vrp` 文件**不复制进评测沙箱**：评测器从宿主（`FRONTIER_EVAL_UNIFIED_SOURCE_BENCHMARK_DIR`）读取路径，只在评分该实例时把路径交给候选。这些文件在仓库里是公开的，运行中的候选收到路径后确实能读到文件（见下方威胁模型）；`Task.md` 和 `constraints.txt` 会告知 agent 存在这类 held-out 实例且会被评分，但代码生成时拿不到它们的实例数据，因此 LLM 无法按实例名硬编码路线；`validator.py` 还会静态拒绝按实例名硬编码，`CVRP_EVAL_GENERATE_SEED` 则让评分时的实例集不可预测。
+- **评测时生成实例**：设置 `CVRP_EVAL_GENERATE_SEED`（可选 `CVRP_EVAL_GENERATE_COUNT`，默认 6），评测器会在评分时用该种子**现场生成全新实例**参与评分，每个生成实例的参考距离由参考求解器当场计算——这样即使候选见过所有公开实例文件，也无法背答案。同一种子 ⇒ 同一批实例 ⇒ 完全可复现。直跑评测器与 unified 运行时（process 模式）均支持；docker 隔离模式（Linux/WSL）下评分可用（靠 `eval_command.txt` 的 `{benchmark_source}` 注入），但运行时生成不可用——unified 运行时不会把种子环境变量传入容器（框架级限制）。
+- **沙箱隔离**：`copy_files.txt` 只把 `baseline/`、`data/instances/`、`frontier_eval/` 复制进评测沙箱（held-out 实例与 `reference.json` 从宿主读取、从不复制）。`frontier_eval/evaluator.py` 是自包含的（解析、校验、评分与完整性检查全部内嵌），因此**任何 `verification/` 文件（包括参考求解器）都不复制**。`reference.json` 从不复制；评测器通过 `FRONTIER_EVAL_UNIFIED_SOURCE_BENCHMARK_DIR` 从宿主读取参考距离，候选子进程运行时**所有 `FRONTIER_*` 变量都被剥离**（外加参考/生成设置），无法得知宿主仓库路径——这封死了 `FRONTIER_ENGINEERING_ROOT` 侧信道（否则候选可借此找到并导入宿主上的 `verification/ref_solver.py`）。
+- **Preflight 检查**（`verification/validator.py`）：评测器会静态拒绝修改 EVOLVE-BLOCK 区外代码、引用 `verification` / `ref_solver` / `reference.json`、含绝对路径、或按实例名硬编码路线的候选；另有**确定性探针**——在最小 / 中等 / 最大 3 个代表实例上把候选各跑两次，输出不一致（非确定性）即判无效。
+- **威胁模型**：所有实例文件（公开与 held-out）在仓库和评测沙箱里都可见——能读到仓库的人总能手写一个求解器，任何 benchmark 都无法阻止这一点。防护是威慑级的：constraints 禁止读取参考与按实例名硬编码，`validator.py` 静态拒绝此类尝试，`CVRP_EVAL_GENERATE_SEED` 让评分时的实例集不可预测。unified 运行时（process 与 docker 隔离）会把宿主仓库暴露给候选进程（这是所有任务共享的框架级行为）。评分只衡量解的质量，从不看代码来源。
+
+## 实验记录
+
+所有 agent 运行均使用 `deepseek-v4-flash` 模型。分数为 "best found"；LLM 进化具有随机性，有多次运行的地方一并列出。
+
+| 运行 | 框架 | 分数 | 评测集 |
+|------|------|------|--------|
+| baseline（随机顺序最近插入） | — | 54.69 | 24 实例（12 公开 + 12 held-out） |
+| reference（GRASP + LNS，评分基准） | — | 100 | 24 实例 |
+| `runs/.../openevolve/deepseek-v4-flash/20260807_170244` | openevolve，5 轮 | 96.38 | 12 公开 |
+| `runs/.../openevolve/deepseek-v4-flash/20260807_195321` | openevolve，5 轮 | 95.65 | 12 公开 |
+| `runs/.../openevolve/deepseek-v4-flash/20260811_215703` | openevolve，5 轮 | 98.00 | 24 实例（12 公开 + 12 held-out） |
+| `runs/.../openevolve/deepseek-v4-flash/20260812_172944` | openevolve，5 轮 | 97.96 | 24 实例（12 公开 + 12 held-out） |
+| `runs/.../openevolve/deepseek-v4-flash/20260812_174122` | openevolve，5 轮 | 97.47 | 24 实例（12 公开 + 12 held-out） |
+| `runs/.../shinkaevolve/deepseek-v4-flash/20260807_195503` | ShinkaEvolve，5 代 | 99.31 | 12 公开 |
+| `runs/.../shinkaevolve/deepseek-v4-flash/20260811_192446` | ShinkaEvolve，5 代 | 98.13 | 24 实例（12 公开 + 12 held-out） |
+| `runs/.../shinkaevolve/deepseek-v4-flash/20260812_175208` | ShinkaEvolve，5 代 | 54.69\* | 24 实例（12 公开 + 12 held-out） |
+| `runs/.../shinkaevolve/deepseek-v4-flash/20260812_175816` | ShinkaEvolve，5 代 | 98.65 | 24 实例（12 公开 + 12 held-out） |
+| `runs/.../abmcts/deepseek-v4-flash/20260807_190515` | AB-MCTS，5 候选 | 98.70 | 12 公开 |
+| `runs/.../abmcts/deepseek-v4-flash/20260812_102741` | AB-MCTS，5 候选 | 98.49 | 24 实例（12 公开 + 12 held-out） |
+| `runs/.../abmcts/deepseek-v4-flash/20260812_181308` | AB-MCTS，5 候选 | 96.78 | 24 实例（12 公开 + 12 held-out） |
+| `runs/.../abmcts/deepseek-v4-flash/20260812_182222` | AB-MCTS，5 候选 | 99.26 | 24 实例（12 公开 + 12 held-out） |
+
+baseline 复现：
+
+```bash
+# 在 CVRP 目录下
+python verification/evaluator.py baseline/solver.py # -> 54.69, valid 1.0（24 实例）
+python verification/test_evaluator.py # -> 23 个单元测试全部通过
+python verification/test_validator.py # -> 16 个单元测试全部通过
+python verification/test_ref_solver.py # -> 5 个单元测试全部通过
+python verification/test_frontier_eval_evaluator.py # -> 6 个单元测试全部通过（沙箱版）
+python verification/multiseed_stat.py --seeds 111 222 333 # -> 57.73 ± 1.58
+```
+
+框架适配验证（仓库根目录）：
+
+```bash
+# process 模式
+python -m frontier_eval task=unified task.benchmark=VehicleRouting/CVRP algorithm.iterations=0
+
+# docker 隔离（先构建镜像：docker build -t cvrp-benchmark -f verification/docker/Dockerfile .）
+python -m frontier_eval task=unified task.benchmark=VehicleRouting/CVRP algorithm.iterations=0 task.runtime.isolation_mode=docker task.runtime.docker_image=cvrp-benchmark
+```
+
+## Unified-task 集成
+
+- **Benchmark id**：`VehicleRouting/CVRP`
+- 评测器仅使用 Python 标准库，因此**无需任何 runtime 覆盖项**（不需要 `python_path`、conda 环境或 Docker 镜像指定）；`eval_command.txt` 直接使用 `{python}`。
+- 仅 Windows：本地验证需将 `task.runtime.shell` 指向 Git Bash（如 `task.runtime.shell=C:/Program Files/Git/bin/bash.exe`），因为默认 `bash` 会解析到没有 `python` 的 WSL 垫片（returncode 127）。Linux 无需覆盖。
+- 框架适配验证命令（仓库根目录）：
+ `python -m frontier_eval task=unified task.benchmark=VehicleRouting/CVRP algorithm.iterations=0`
diff --git a/benchmarks/VehicleRouting/CVRP/Task.md b/benchmarks/VehicleRouting/CVRP/Task.md
new file mode 100644
index 00000000..1dab1cb5
--- /dev/null
+++ b/benchmarks/VehicleRouting/CVRP/Task.md
@@ -0,0 +1,193 @@
+# Task: Capacitated Vehicle Routing Problem (CVRP)
+
+## Audience and assumptions
+
+This task assumes a general CS background but no (or little) prior exposure to
+combinatorial optimization / vehicle routing.
+
+## Background
+
+In logistics, trucks depart from a warehouse to deliver goods to customers. The
+**capacitated vehicle routing problem (CVRP)** is its standard mathematical
+model:
+
+- **Depot**: node 0; every vehicle starts and ends here.
+- **Customers**: nodes 1..n, each with a demand `demand[c]`.
+- **Vehicles**: identical, capacity limit `capacity`.
+- **Route**: a sequence `depot → some customers → depot` whose total demand
+ does not exceed `capacity`.
+
+Goal: serve **all** customers (each exactly once) with any number of routes,
+minimizing **total travel distance**.
+
+CVRP is NP-hard: instances with a few dozen customers cannot be solved exactly
+and require heuristics (nearest-neighbour, savings, 2-opt, large neighbourhood
+search, metaheuristics, ...).
+
+## Instances (data/instances/, data/instances_heldout/)
+
+24 deterministically generated clustered instances (modelling city-like
+customer distributions, coordinates 1..100, rounded Euclidean distances):
+
+- **12 public instances** (data/instances/, `VRP-*`):
+
+| Instance | Customers | Capacity | Instance | Customers | Capacity |
+|----------|-----------|----------|----------|-----------|----------|
+| VRP-19-2 | 19 | 270 | VRP-45-7 | 45 | 150 |
+| VRP-21-3 | 21 | 165 | VRP-48-7 | 48 | 170 |
+| VRP-22-4 | 22 | 130 | VRP-54-8 | 54 | 175 |
+| VRP-32-5 | 32 | 160 | VRP-55-8 | 55 | 185 |
+| VRP-37-6 | 37 | 145 | VRP-60-9 | 60 | 160 |
+| VRP-45-6 | 45 | 190 | VRP-60-10 | 60 | 165 |
+
+- **12 held-out instances** (data/instances_heldout/, `VHO-*`): scored at
+ evaluation time alongside the public ones, doubling the evaluation set to 24.
+ Their files stay on the host and are **not** copied into the evaluation
+ sandbox, so you cannot read them during development — each path is handed to
+ your solver only at scoring time. Hardcoding routes by instance name is
+ rejected statically, and `CVRP_EVAL_GENERATE_SEED` can add fresh instances at
+ evaluation time so the scored set is not predictable. Scoring on the
+ held-out set measures whether the agent learned a *generalizable* solving
+ method.
+
+The file name is the instance name (e.g. `VRP-19-2.vrp`), in TSPLIB style
+(`NODE_COORD_SECTION` / `DEMAND_SECTION` / `DEPOT_SECTION`, depot is node 1).
+Instances are generated deterministically by `verification/generate_instances.py`
+(seed 42; the per-instance `seed_key` is fixed to the original release
+identifiers so the dataset stays byte-identical across releases).
+
+## Input / output contract
+
+### Candidate program `baseline/solver.py`
+
+```python
+# EVOLVE-BLOCK-START
+def solve(instance):
+ """Takes an instance dict, returns a list of routes list[list[int]].
+ Each route is a visit sequence of customer ids (1..n, depot 0 excluded)."""
+ ...
+# EVOLVE-BLOCK-END
+```
+
+- `instance` fields:
+ - `n`: number of customers (ids 1..n; depot is 0)
+ - `capacity`: vehicle capacity
+ - `demand`: `demand[0..n]`, `demand[0] == 0`
+ - `distance`: `(n+1)×(n+1)` rounded-Euclidean distance matrix,
+ `distance[0][c]` = depot-to-customer-c distance
+- Output: `list[list[int]]`. Each route is a **customer-id sequence** (depot 0
+ excluded), e.g. `[[3,1,5],[2,4]]` means two vehicles.
+- Standalone run: `python baseline/solver.py `
+ (the fixed I/O part must not be modified).
+
+### Validation rules
+
+Each candidate output is checked:
+1. Well-formed: `routes` is a list of lists of integers in 1..n;
+2. **Full coverage**: the union of all routes is exactly {1..n}
+ (no duplicates, none missing);
+3. **Capacity**: `sum(demand[c]) <= capacity` per route.
+
+Any violation → that instance is invalid (0 points) and the whole candidate
+gets `valid=0`.
+
+### Candidate integrity checks (preflight)
+
+Before running, the evaluator statically rejects a candidate that:
+- removes or reorders the `EVOLVE-BLOCK-START` / `EVOLVE-BLOCK-END` markers,
+ or modifies code outside the evolve block relative to the initial baseline;
+- references the verification module, the reference solver, or
+ `reference.json` (e.g. `import verification.ref_solver`);
+- contains absolute filesystem paths;
+- hardcodes per-instance routes by name (e.g. `"VRP-19-2": [...]`).
+
+Violating candidates score 0 and are marked invalid. The candidate subprocess
+also runs in an environment stripped of host benchmark paths, so it cannot
+locate `reference.json` on the host; the evaluation sandbox contains only the
+files the candidate needs (instances + evaluator glue), never the reference
+solver or `reference.json`.
+
+## Scoring
+
+```
+score_instance = min(100, 100 × reference_distance / candidate_distance)
+combined_score = mean(score_instance) # average over the 24 instances (12 public + 12 held-out)
+valid = all instances valid ? 1 : 0
+```
+
+- `reference_distance` comes from `data/reference.json`, precomputed by
+ `verification/ref_solver.py`: a **deterministic** GRASP multi-start +
+ 2-opt + relocate/swap/2-opt* + LNS (greedy repair with tabu diversification)
+ that keeps the best result over the fixed seed list `(123, 2024, 7)`, so
+ regeneration is byte-identical on any machine. The reference is
+ near-optimal (cross-checked against the archived best agent solver and an
+ OR-Tools GLS solve).
+- **100 = reference-quality solutions**; candidates shorter than the reference
+ exceed 100 and are truncated to 100. Reaching 100 means the solution is
+ close to the practical optimum of the instance.
+- Optional score knob `CVRP_EVAL_SCORE_SCALE` (default 1.0):
+ `score = min(100, scale × 100 × ref / cand)`. A scale < 1 tightens the 100
+ bar (e.g. scale=2/3 requires the candidate to be no longer than 2/3 of the
+ reference distance to score 100).
+- Invalid / crashing / timing-out candidates: 0 on that instance and
+ `valid=0` for the whole run.
+
+## How to run
+
+```bash
+# Evaluate a candidate solver (inside the CVRP directory)
+python verification/evaluator.py baseline/solver.py
+
+# Evaluate on a subset of instances
+python verification/evaluator.py baseline/solver.py --instances VRP-19-2 VRP-32-5
+
+# Run the unit tests (evaluator / validator / candidate checks)
+python verification/test_evaluator.py
+
+# Unified-task adapter check (repo root, process mode)
+python -m frontier_eval task=unified task.benchmark=VehicleRouting/CVRP algorithm.iterations=0
+
+# Unified-task adapter check (repo root, docker isolation; requires building
+# the image first: docker build -t cvrp-benchmark -f verification/docker/Dockerfile .)
+python -m frontier_eval task=unified task.benchmark=VehicleRouting/CVRP algorithm.iterations=0 task.runtime.isolation_mode=docker task.runtime.docker_image=cvrp-benchmark
+```
+
+Environment variables: `CVRP_EVAL_TIMEOUT_S` (per-instance subprocess timeout,
+default 60), `CVRP_EVAL_INSTANCES` (instance subset), `CVRP_EVAL_MAX_INSTANCES`
+(instance cap), `CVRP_EVAL_SCORE_SCALE` (score knob, default 1.0).
+
+## Reference scores (measured on this machine, against the current reference.json)
+
+The current evaluation set is 24 instances (12 public + 12 held-out). The
+agent scores below were measured on the **earlier 12-public-instance set**
+(held-out instances were added later) and are kept for cross-framework
+comparison; fresh runs on the full 24-instance set (ShinkaEvolve 98.65,
+openevolve 98.00, AB-MCTS 99.26) show the learned solvers generalize to
+unseen instances. See README "Experiments" for run records and multi-run
+statistics.
+
+| Solver | combined_score |
+|--------|----------------|
+| baseline (random-order cheapest insertion), 24 instances | **54.69** |
+| reference (deterministic GRASP + LNS, scoring baseline) | 100 (near-optimal) |
+| agent (openevolve, 5 iterations, best, 12-instance set) | 96.38 |
+| agent (openevolve, 5 iterations, best, 24-instance set) | 98.00 |
+| agent (ShinkaEvolve, 5 generations, best, 12-instance set) | 99.31 |
+| agent (ShinkaEvolve, 5 generations, best, 24-instance set) | 98.65 |
+| agent (AB-MCTS, 5 candidates, best, 12-instance set) | 98.70 |
+| agent (AB-MCTS, 5 candidates, best, 24-instance set) | 99.26 |
+
+## Optimisation hints (weak → strong)
+
+1. **Random-order cheapest insertion** (baseline): insert customers one by
+ one, in a random order, at their cheapest feasible position across all
+ routes — a weak but standard randomized construction.
+2. **Clarke-Wright savings**: merge routes by decreasing
+ `d(0,i)+d(0,j)-d(i,j)` — a clear improvement.
+3. **Intra-route 2-opt**: flip route segments to remove crossings — better.
+4. **Cross-route local search**: relocate / swap / 2-opt*.
+5. **Large neighbourhood search (LNS) / simulated annealing / genetic
+ algorithms**: approach or beat the reference.
+
+Every step is a verifiable score gain. **First guarantee validity (full
+coverage + capacity), then optimize distance.**
diff --git a/benchmarks/VehicleRouting/CVRP/Task_zh-CN.md b/benchmarks/VehicleRouting/CVRP/Task_zh-CN.md
new file mode 100644
index 00000000..e369ddf5
--- /dev/null
+++ b/benchmarks/VehicleRouting/CVRP/Task_zh-CN.md
@@ -0,0 +1,137 @@
+# Task：容量约束车辆路径问题（CVRP）
+
+## 受众与假设
+
+本任务假设你有一般 CS 背景，但对组合优化 / 车辆路径问题没有或只有很少了解。
+
+## 问题背景
+
+物流配送中，一辆辆货车从仓库出发为客户送货。**容量约束车辆路径问题（CVRP）** 是它的标准数学模型：
+
+- **仓库（depot）**：编号 0，所有车辆从这里出发并返回。
+- **客户**：编号 1..n，每个客户有需求量 `demand[c]`。
+- **车辆**：同型，容量上限 `capacity`。
+- **路线**：每条路线为 `仓库 → 若干客户 → 仓库`，且路线内总需求 ≤ 容量。
+
+目标：用若干条路线服务**所有**客户（每个客户恰好被服务一次），使**总行驶距离最小**。
+
+CVRP 是 NP-hard 问题，几十个客户的实例无法精确求解，必须使用启发式方法（最近邻、savings、2-opt、大邻域搜索、元启发式等）。
+
+## 实例（data/instances/、data/instances_heldout/）
+
+本任务提供 24 个确定性生成的聚簇分布实例（模拟城市客户分布，坐标 1..100，距离四舍五入取整）：
+
+- **12 个公开实例**（data/instances/，`VRP-*`）：
+
+| 实例 | 客户数 | 容量 | 实例 | 客户数 | 容量 |
+|------|--------|------|------|--------|------|
+| VRP-19-2 | 19 | 270 | VRP-45-7 | 45 | 150 |
+| VRP-21-3 | 21 | 165 | VRP-48-7 | 48 | 170 |
+| VRP-22-4 | 22 | 130 | VRP-54-8 | 54 | 175 |
+| VRP-32-5 | 32 | 160 | VRP-55-8 | 55 | 185 |
+| VRP-37-6 | 37 | 145 | VRP-60-9 | 60 | 160 |
+| VRP-45-6 | 45 | 190 | VRP-60-10 | 60 | 165 |
+
+- **12 个 held-out 实例**（data/instances_heldout/，`VHO-*`）：与公开实例一起在评测时打分，把评测集扩充到 24 个。它们的文件**留在宿主、不复制进评测沙箱**，开发阶段你读不到它们——评分时才把每个路径交给你的求解器。按实例名硬编码会被静态检查拒绝，`CVRP_EVAL_GENERATE_SEED` 还可在评测时现场加入新实例，让评分实例集不可预测。在 held-out 集上打分衡量 agent 是否学到了**可泛化**的求解方法。
+
+文件名即实例名（如 `VRP-19-2.vrp`），采用 TSPLIB 风格格式（`NODE_COORD_SECTION` / `DEMAND_SECTION` / `DEPOT_SECTION`，depot 为节点 1）。实例由 `verification/generate_instances.py` 确定性生成（seed 42，`seed_key` 固定为发布时的原始标识符，保证数据集跨版本逐字节稳定）。
+
+## 输入 / 输出契约
+
+### 候选程序 `baseline/solver.py`
+
+```python
+# EVOLVE-BLOCK-START
+def solve(instance):
+ """输入 instance dict，输出路线列表 list[list[int]]。
+ 每个子列表是一条路线的客户访问序列（客户编号 1..n，不含仓库 0）。"""
+ ...
+# EVOLVE-BLOCK-END
+```
+
+- `instance` 字段：
+ - `n`：客户数（客户编号 1..n，仓库为 0）
+ - `capacity`：车辆容量
+ - `demand`：`demand[0..n]`，`demand[0] == 0`
+ - `distance`：`(n+1)×(n+1)` 的欧氏四舍五入取整距离矩阵，`distance[0][c]` 为仓库到客户 c 的距离
+- 输出：`list[list[int]]`。每条路线为**客户编号序列**（不含仓库 0），例如 `[[3,1,5],[2,4]]` 表示两辆车。
+- 独立运行：`python baseline/solver.py `（固定 I/O 部分不可修改）。
+
+### 验证规则
+
+对候选输出逐条检查：
+1. 格式合法：`routes` 是列表的列表，元素为 1..n 的整数；
+2. **全覆盖**：所有路线的客户并集恰好为 {1..n}（无重复、无遗漏）；
+3. **容量**：每条路线 `sum(demand[c]) ≤ capacity`。
+
+任一不满足 → 该实例判 invalid，得 0 分，且整个候选 `valid=0`。
+
+### 候选完整性检查（preflight）
+
+运行前，评测器会**静态拒绝**有以下行为的候选：
+- 删除或重排 `EVOLVE-BLOCK-START` / `EVOLVE-BLOCK-END` 标记，或修改 evolve 区外、与初始 baseline 不一致的代码；
+- 引用 verification 模块、参考求解器或 `reference.json`（如 `import verification.ref_solver`）；
+- 含绝对文件系统路径；
+- 按实例名硬编码路线（如 `"VRP-19-2": [...]`）。
+
+违规候选得 0 分并判无效。候选子进程运行在剥离宿主路径的环境中，无法定位宿主上的 `reference.json`；评测沙箱只包含候选需要的文件（实例 + 评测胶水），**从不包含参考求解器或 `reference.json`**。
+
+## 评分
+
+```
+score_instance = min(100, 100 × reference_distance / candidate_distance)
+combined_score = mean(score_instance) # 跨 24 个实例平均（12 公开 + 12 held-out）
+valid = 全部实例合法？1 : 0
+```
+
+- `reference_distance` 来自 `data/reference.json`，由 `verification/ref_solver.py` 预计算：**确定性**的 GRASP 多起点 + 2-opt + relocate/swap + 2-opt* + LNS（贪心修复 + tabu 多样化），对固定种子列表 `(123, 2024, 7)` 逐实例取最优，任何机器上重新生成均字节一致。参考解为近最优（与本任务存档的 agent 最优解、OR-Tools GLS 交叉验证一致）。
+- **100 分 = 达到参考求解器的解质量**；候选解比参考解更短时可超过 100 分（被截断在 100）。达到 100 意味着已经逼近该实例的实际最优。
+- 可选评分旋钮 `CVRP_EVAL_SCORE_SCALE`（默认 1.0）：`score = min(100, scale × 100 × ref / cand)`。scale < 1 会收紧 100 分线（如 scale=2/3 要求候选距离 ≤ 参考的 2/3 才能得 100）。
+- 非法 / 崩溃 / 超时的候选：对应实例 0 分，且 `valid=0`（整体判负）。
+
+## 如何运行
+
+```bash
+# 在 CVRP 目录下评测一个候选求解器
+python verification/evaluator.py baseline/solver.py
+
+# 只评测部分实例
+python verification/evaluator.py baseline/solver.py --instances VRP-19-2 VRP-32-5
+
+# 运行单元测试（评测器 / 验证器 / 候选检查）
+python verification/test_evaluator.py
+
+# 框架适配验证（仓库根目录，process 模式）
+python -m frontier_eval task=unified task.benchmark=VehicleRouting/CVRP algorithm.iterations=0
+
+# 框架适配验证（仓库根目录，docker 隔离；需先构建镜像：
+# docker build -t cvrp-benchmark -f verification/docker/Dockerfile .）
+python -m frontier_eval task=unified task.benchmark=VehicleRouting/CVRP algorithm.iterations=0 task.runtime.isolation_mode=docker task.runtime.docker_image=cvrp-benchmark
+```
+
+环境变量：`CVRP_EVAL_TIMEOUT_S`（每实例子进程超时，默认 60）、`CVRP_EVAL_INSTANCES`（实例子集）、`CVRP_EVAL_MAX_INSTANCES`（实例数上限）、`CVRP_EVAL_SCORE_SCALE`（评分旋钮，默认 1.0）。
+
+## 参考分数（本机实测，对当前 reference.json）
+
+当前评测集为 24 个实例（12 公开 + 12 held-out）。下面的 agent 分数是在**较早的 12 公开实例集**上测得的（held-out 实例为后加），保留用于跨框架对比；在完整 24 实例集上的新运行（ShinkaEvolve 98.65、openevolve 98.00、AB-MCTS 99.26）证明学到的求解器能泛化到未见过的实例。运行记录与多运行统计见 README "Experiments"。
+
+| 求解器 | combined_score |
+|--------|----------------|
+| baseline（随机顺序最近插入），24 实例 | **54.69** |
+| reference（确定性 GRASP + LNS，评分基准） | 100（近最优） |
+| agent（openevolve 5 轮，best，12 实例集） | 96.38 |
+| agent（openevolve 5 轮，best，24 实例集） | 98.00 |
+| agent（ShinkaEvolve 5 代，best，12 实例集） | 99.31 |
+| agent（ShinkaEvolve 5 代，best，24 实例集） | 98.65 |
+| agent（AB-MCTS 5 候选，best，12 实例集） | 98.70 |
+| agent（AB-MCTS 5 候选，best，24 实例集） | 99.26 |
+
+## 优化提示（由弱到强）
+
+1. **随机顺序最近插入**（baseline）：按随机顺序逐个把客户插入到所有路线中代价最小的可行位置 —— 弱但标准的随机化构造。
+2. **Clarke-Wright savings**：按 `d(0,i)+d(0,j)-d(i,j)` 降序合并路线 —— 明显改进。
+3. **路线内 2-opt**：翻转路线片段消除交叉 —— 进一步改进。
+4. **跨路线局部搜索**：relocate / swap / 2-opt*。
+5. **大邻域搜索（LNS）/ 模拟退火 / 遗传**：接近参考甚至超越参考。
+
+每一步都是可验证的分数提升。**先保证合法（全覆盖 + 容量），再优化距离。**
diff --git a/benchmarks/VehicleRouting/CVRP/baseline/result_log.txt b/benchmarks/VehicleRouting/CVRP/baseline/result_log.txt
new file mode 100644
index 00000000..1eb63a26
--- /dev/null
+++ b/benchmarks/VehicleRouting/CVRP/baseline/result_log.txt
@@ -0,0 +1,61 @@
+CVRP baseline evaluation log
+=============================
+Solver : baseline/solver.py (random-order cheapest insertion, seed 42)
+Evaluator : verification/evaluator.py (stdlib only)
+Date : 2026-08-10
+Command: python verification/evaluator.py baseline/solver.py
+
+combined_score: 54.69
+valid: 1.0
+ OK VRP-19-2: score=68.00
+ OK VRP-21-3: score=71.55
+ OK VRP-22-4: score=66.17
+ OK VRP-32-5: score=49.19
+ OK VRP-37-6: score=60.18
+ OK VRP-45-6: score=55.47
+ OK VRP-45-7: score=57.07
+ OK VRP-48-7: score=48.49
+ OK VRP-54-8: score=48.45
+ OK VRP-55-8: score=47.15
+ OK VRP-60-10: score=45.67
+ OK VRP-60-9: score=49.67
+ OK VHO-22-3: score=51.85
+ OK VHO-24-3: score=56.64
+ OK VHO-27-4: score=58.49
+ OK VHO-29-4: score=52.86
+ OK VHO-33-5: score=59.13
+ OK VHO-36-5: score=56.53
+ OK VHO-39-6: score=63.98
+ OK VHO-43-6: score=54.66
+ OK VHO-47-7: score=53.07
+ OK VHO-51-7: score=44.03
+ OK VHO-55-8: score=50.00
+ OK VHO-58-9: score=44.27
+wall time: 2.5s
+
+Notes
+-----
+* 24 instances = 12 public (data/instances/, VRP-*) + 12 held-out
+ (data/instances_heldout/, VHO-*). Held-out instances are scored at
+ evaluation time only and never shown to agents.
+* Reference distances are precomputed by verification/ref_solver.py
+ (deterministic, seeds (123, 2024, 7)) into data/reference.json.
+* The candidate subprocess runs without FRONTIER_EVAL_UNIFIED_* and
+ reference-distance env vars, and the evaluator statically rejects
+ candidates that reference verification code, reference.json, absolute
+ paths, or per-instance hardcoded routes (see check_candidate).
+
+Multi-seed statistics (verification/multiseed_stat.py)
+-------------------------------------------------------
+The released evaluation set is fixed, so the (deterministic) baseline scores
+identically on it. To report a multi-seed distribution, this script derives
+fresh instance sets from three seeds (generate_instances generate_instance
+seed=...) and computes each set's reference distances on the fly.
+
+Command: python verification/multiseed_stat.py --seeds 111 222 333
+
+seed 111: combined=57.24
+seed 222: combined=59.86
+seed 333: combined=56.09
+multi-seed: mean=57.73 std=1.58 min=56.09 max=59.86 seeds=[111, 222, 333]
+
diff --git a/benchmarks/VehicleRouting/CVRP/baseline/solver.py b/benchmarks/VehicleRouting/CVRP/baseline/solver.py
new file mode 100644
index 00000000..cb2e2cbf
--- /dev/null
+++ b/benchmarks/VehicleRouting/CVRP/baseline/solver.py
@@ -0,0 +1,130 @@
+"""CVRP candidate solver (baseline: random-order cheapest insertion).
+
+Contract:
+ * Run standalone: python baseline/solver.py
+ * `solve(instance)` returns a list of routes; each route is a list of
+ customer ids (1..n) visited in order. The depot (id 0) is implicit at
+ both ends and must NOT appear in the route.
+ * instance dict fields:
+ - n : number of customers (ids 1..n)
+ - capacity : vehicle capacity
+ - demand : demand[0..n], demand[0] == 0
+ - distance : (n+1)x(n+1) rounded Euclidean distance matrix
+"""
+import json
+import math
+import sys
+from pathlib import Path
+
+# ---------------------------------------------------------------------------
+# Fixed section: instance parsing and I/O. Do not modify.
+# ---------------------------------------------------------------------------
+def parse_instance(path):
+ """Parse a TSPLIB-style CVRP .vrp file into an instance dict."""
+ text = Path(path).read_text(encoding="utf-8", errors="ignore")
+ coords = {}
+ demands = {}
+ capacity = 0
+ section = None
+ for line in text.splitlines():
+ line = line.strip()
+ if not line:
+ continue
+ upper = line.upper()
+ if upper.startswith("CAPACITY"):
+ capacity = int(line.split(":")[-1].strip())
+ continue
+ if upper == "NODE_COORD_SECTION":
+ section = "coords"
+ continue
+ if upper == "DEMAND_SECTION":
+ section = "demand"
+ continue
+ if upper == "DEPOT_SECTION":
+ section = None
+ continue
+ if upper == "EOF" or upper.startswith(("EDGE_WEIGHT", "DISPLAY_DATA")):
+ section = None
+ continue
+ if upper.startswith(("NAME", "COMMENT", "TYPE", "DIMENSION")):
+ continue
+ if section == "coords":
+ parts = line.split()
+ if len(parts) >= 3:
+ coords[int(parts[0])] = (float(parts[1]), float(parts[2]))
+ elif section == "demand":
+ parts = line.split()
+ if len(parts) >= 2:
+ demands[int(parts[0])] = int(parts[1])
+ n_customers = max(coords) - 1 # depot is id 1, customers are ids 2..n+1
+ pts = [coords[1]] + [coords[i] for i in range(2, n_customers + 2)]
+ dist = [[0] * (n_customers + 1) for _ in range(n_customers + 1)]
+ for i in range(n_customers + 1):
+ for j in range(n_customers + 1):
+ dx = pts[i][0] - pts[j][0]
+ dy = pts[i][1] - pts[j][1]
+ dist[i][j] = int(round(math.hypot(dx, dy)))
+ return {
+ "n": n_customers,
+ "capacity": capacity,
+ "demand": [0] + [demands.get(i, 0) for i in range(2, n_customers + 2)],
+ "distance": dist,
+ }
+
+
+def main():
+ inst_path, out_path = sys.argv[1], sys.argv[2]
+ inst = parse_instance(inst_path)
+ routes = solve(inst)
+ with open(out_path, "w", encoding="utf-8") as fh:
+ json.dump(routes, fh)
+
+
+# ---------------------------------------------------------------------------
+# EVOLVE-BLOCK-START
+# ---------------------------------------------------------------------------
+def solve(instance):
+ """Random-order cheapest insertion: process customers in a random order
+ (seeded) and insert each at its cheapest feasible position across all
+ routes. A standard randomized-construction baseline."""
+ import random
+
+ rng = random.Random(42)
+ n = instance["n"]
+ capacity = instance["capacity"]
+ demand = instance["demand"]
+ dist = instance["distance"]
+
+ order = list(range(1, n + 1))
+ rng.shuffle(order)
+ routes = []
+
+ def best_place(cust):
+ best = None
+ for ri, route in enumerate(routes):
+ load = sum(demand[c] for c in route)
+ if load + demand[cust] > capacity:
+ continue
+ for pos in range(len(route) + 1):
+ prev = 0 if pos == 0 else route[pos - 1]
+ nxt = 0 if pos == len(route) else route[pos]
+ cost = dist[prev][cust] + dist[cust][nxt] - dist[prev][nxt]
+ if best is None or cost < best[0]:
+ best = (cost, ri, pos)
+ return best
+
+ for cust in order:
+ b = best_place(cust)
+ if b is None:
+ routes.append([cust])
+ else:
+ routes[b[1]].insert(b[2], cust)
+ return routes
+
+
+# ---------------------------------------------------------------------------
+# EVOLVE-BLOCK-END
+# ---------------------------------------------------------------------------
+
+if __name__ == "__main__":
+ main()
diff --git a/benchmarks/VehicleRouting/CVRP/data/instances/VRP-19-2.vrp b/benchmarks/VehicleRouting/CVRP/data/instances/VRP-19-2.vrp
new file mode 100644
index 00000000..fd8434f7
--- /dev/null
+++ b/benchmarks/VehicleRouting/CVRP/data/instances/VRP-19-2.vrp
@@ -0,0 +1,50 @@
+NAME: VRP-19-2
+COMMENT: generated clustered CVRP instance (deterministic)
+TYPE: CVRP
+DIMENSION: 20
+CAPACITY: 270
+NODE_COORD_SECTION
+1 50.0 50.0
+2 41 41
+3 62 36
+4 31 40
+5 62 23
+6 41 46
+7 59 36
+8 39 39
+9 57 25
+10 31 37
+11 60 49
+12 49 48
+13 62 29
+14 37 55
+15 45 39
+16 29 34
+17 62 29
+18 31 44
+19 50 22
+20 32 56
+DEMAND_SECTION
+1 0
+2 21
+3 32
+4 25
+5 24
+6 28
+7 29
+8 5
+9 10
+10 31
+11 23
+12 33
+13 27
+14 10
+15 13
+16 19
+17 28
+18 32
+19 26
+20 14
+DEPOT_SECTION
+1
+EOF
diff --git a/benchmarks/VehicleRouting/CVRP/data/instances/VRP-21-3.vrp b/benchmarks/VehicleRouting/CVRP/data/instances/VRP-21-3.vrp
new file mode 100644
index 00000000..7e6500e8
--- /dev/null
+++ b/benchmarks/VehicleRouting/CVRP/data/instances/VRP-21-3.vrp
@@ -0,0 +1,54 @@
+NAME: VRP-21-3
+COMMENT: generated clustered CVRP instance (deterministic)
+TYPE: CVRP
+DIMENSION: 22
+CAPACITY: 165
+NODE_COORD_SECTION
+1 50.0 50.0
+2 82 53
+3 63 40
+4 49 40
+5 84 48
+6 82 44
+7 54 58
+8 90 64
+9 72 45
+10 51 44
+11 79 51
+12 66 34
+13 38 33
+14 87 58
+15 71 41
+16 52 63
+17 81 58
+18 76 26
+19 41 57
+20 75 57
+21 72 42
+22 50 47
+DEMAND_SECTION
+1 0
+2 19
+3 9
+4 27
+5 17
+6 22
+7 12
+8 33
+9 19
+10 22
+11 8
+12 25
+13 17
+14 7
+15 7
+16 21
+17 12
+18 27
+19 6
+20 16
+21 32
+22 35
+DEPOT_SECTION
+1
+EOF
diff --git a/benchmarks/VehicleRouting/CVRP/data/instances/VRP-22-4.vrp b/benchmarks/VehicleRouting/CVRP/data/instances/VRP-22-4.vrp
new file mode 100644
index 00000000..3c3305e7
--- /dev/null
+++ b/benchmarks/VehicleRouting/CVRP/data/instances/VRP-22-4.vrp
@@ -0,0 +1,56 @@
+NAME: VRP-22-4
+COMMENT: generated clustered CVRP instance (deterministic)
+TYPE: CVRP
+DIMENSION: 23
+CAPACITY: 130
+NODE_COORD_SECTION
+1 50.0 50.0
+2 35 33
+3 11 64
+4 71 29
+5 48 33
+6 17 61
+7 77 28
+8 60 27
+9 25 60
+10 74 31
+11 44 41
+12 30 60
+13 61 34
+14 58 31
+15 14 54
+16 80 39
+17 42 24
+18 17 50
+19 76 43
+20 41 20
+21 9 58
+22 59 34
+23 36 30
+DEMAND_SECTION
+1 0
+2 10
+3 16
+4 5
+5 21
+6 33
+7 20
+8 13
+9 21
+10 16
+11 22
+12 25
+13 7
+14 33
+15 8
+16 17
+17 28
+18 23
+19 11
+20 14
+21 22
+22 28
+23 13
+DEPOT_SECTION
+1
+EOF
diff --git a/benchmarks/VehicleRouting/CVRP/data/instances/VRP-32-5.vrp b/benchmarks/VehicleRouting/CVRP/data/instances/VRP-32-5.vrp
new file mode 100644
index 00000000..6cf849ec
--- /dev/null
+++ b/benchmarks/VehicleRouting/CVRP/data/instances/VRP-32-5.vrp
@@ -0,0 +1,76 @@
+NAME: VRP-32-5
+COMMENT: generated clustered CVRP instance (deterministic)
+TYPE: CVRP
+DIMENSION: 33
+CAPACITY: 160
+NODE_COORD_SECTION
+1 50.0 50.0
+2 29 39
+3 71 88
+4 59 69
+5 60 26
+6 42 28
+7 79 60
+8 61 48
+9 73 10
+10 49 45
+11 79 81
+12 57 58
+13 67 16
+14 34 46
+15 71 79
+16 41 61
+17 71 13
+18 44 39
+19 83 76
+20 51 56
+21 63 3
+22 33 44
+23 68 67
+24 59 56
+25 59 14
+26 21 32
+27 78 85
+28 72 54
+29 63 5
+30 56 40
+31 78 75
+32 53 61
+33 66 16
+DEMAND_SECTION
+1 0
+2 31
+3 34
+4 28
+5 18
+6 8
+7 17
+8 13
+9 32
+10 15
+11 27
+12 27
+13 5
+14 15
+15 18
+16 26
+17 11
+18 27
+19 23
+20 26
+21 21
+22 8
+23 20
+24 27
+25 8
+26 35
+27 9
+28 11
+29 29
+30 11
+31 25
+32 9
+33 10
+DEPOT_SECTION
+1
+EOF
diff --git a/benchmarks/VehicleRouting/CVRP/data/instances/VRP-37-6.vrp b/benchmarks/VehicleRouting/CVRP/data/instances/VRP-37-6.vrp
new file mode 100644
index 00000000..5a49908c
--- /dev/null
+++ b/benchmarks/VehicleRouting/CVRP/data/instances/VRP-37-6.vrp
@@ -0,0 +1,86 @@
+NAME: VRP-37-6
+COMMENT: generated clustered CVRP instance (deterministic)
+TYPE: CVRP
+DIMENSION: 38
+CAPACITY: 145
+NODE_COORD_SECTION
+1 50.0 50.0
+2 72 47
+3 78 32
+4 50 89
+5 67 95
+6 71 55
+7 84 35
+8 57 83
+9 63 81
+10 81 49
+11 86 30
+12 64 74
+13 63 75
+14 74 58
+15 74 26
+16 48 73
+17 64 84
+18 74 59
+19 70 32
+20 48 73
+21 68 80
+22 85 60
+23 71 25
+24 48 89
+25 70 67
+26 88 56
+27 87 23
+28 43 72
+29 72 81
+30 73 65
+31 72 30
+32 58 72
+33 59 90
+34 91 55
+35 79 25
+36 58 86
+37 56 83
+38 72 58
+DEMAND_SECTION
+1 0
+2 22
+3 18
+4 14
+5 27
+6 9
+7 22
+8 16
+9 12
+10 12
+11 26
+12 15
+13 11
+14 11
+15 26
+16 24
+17 10
+18 23
+19 25
+20 29
+21 35
+22 15
+23 6
+24 16
+25 26
+26 6
+27 20
+28 10
+29 18
+30 27
+31 30
+32 30
+33 10
+34 13
+35 8
+36 12
+37 30
+38 23
+DEPOT_SECTION
+1
+EOF
diff --git a/benchmarks/VehicleRouting/CVRP/data/instances/VRP-45-6.vrp b/benchmarks/VehicleRouting/CVRP/data/instances/VRP-45-6.vrp
new file mode 100644
index 00000000..c966f74c
--- /dev/null
+++ b/benchmarks/VehicleRouting/CVRP/data/instances/VRP-45-6.vrp
@@ -0,0 +1,102 @@
+NAME: VRP-45-6
+COMMENT: generated clustered CVRP instance (deterministic)
+TYPE: CVRP
+DIMENSION: 46
+CAPACITY: 190
+NODE_COORD_SECTION
+1 50.0 50.0
+2 70 68
+3 48 16
+4 46 29
+5 55 53
+6 54 29
+7 72 79
+8 54 40
+9 45 8
+10 71 51
+11 52 35
+12 69 67
+13 43 28
+14 43 28
+15 73 47
+16 59 45
+17 79 88
+18 47 36
+19 26 20
+20 56 51
+21 57 43
+22 84 70
+23 43 35
+24 31 29
+25 44 44
+26 45 55
+27 84 72
+28 43 39
+29 37 13
+30 70 36
+31 52 46
+32 62 81
+33 39 32
+34 34 29
+35 65 43
+36 65 56
+37 95 69
+38 37 35
+39 39 15
+40 46 40
+41 49 53
+42 84 84
+43 44 27
+44 37 22
+45 52 50
+46 43 46
+DEMAND_SECTION
+1 0
+2 25
+3 35
+4 15
+5 9
+6 9
+7 29
+8 6
+9 13
+10 19
+11 23
+12 13
+13 30
+14 14
+15 33
+16 5
+17 28
+18 5
+19 26
+20 18
+21 9
+22 5
+23 35
+24 10
+25 6
+26 14
+27 9
+28 22
+29 9
+30 23
+31 34
+32 29
+33 28
+34 29
+35 24
+36 33
+37 34
+38 9
+39 19
+40 17
+41 6
+42 35
+43 29
+44 11
+45 25
+46 34
+DEPOT_SECTION
+1
+EOF
diff --git a/benchmarks/VehicleRouting/CVRP/data/instances/VRP-45-7.vrp b/benchmarks/VehicleRouting/CVRP/data/instances/VRP-45-7.vrp
new file mode 100644
index 00000000..5f946984
--- /dev/null
+++ b/benchmarks/VehicleRouting/CVRP/data/instances/VRP-45-7.vrp
@@ -0,0 +1,102 @@
+NAME: VRP-45-7
+COMMENT: generated clustered CVRP instance (deterministic)
+TYPE: CVRP
+DIMENSION: 46
+CAPACITY: 150
+NODE_COORD_SECTION
+1 50.0 50.0
+2 71 26
+3 92 89
+4 39 67
+5 81 46
+6 74 83
+7 74 25
+8 70 75
+9 43 60
+10 71 46
+11 90 91
+12 84 44
+13 80 62
+14 30 64
+15 77 63
+16 77 93
+17 80 36
+18 68 76
+19 35 66
+20 85 46
+21 86 78
+22 65 37
+23 83 74
+24 34 66
+25 78 44
+26 66 85
+27 68 36
+28 83 82
+29 22 73
+30 81 46
+31 76 76
+32 68 33
+33 85 85
+34 34 72
+35 84 46
+36 77 87
+37 70 39
+38 81 85
+39 30 81
+40 79 47
+41 74 74
+42 74 46
+43 86 71
+44 33 65
+45 76 42
+46 71 78
+DEMAND_SECTION
+1 0
+2 6
+3 29
+4 34
+5 30
+6 31
+7 26
+8 26
+9 35
+10 9
+11 31
+12 13
+13 23
+14 5
+15 21
+16 6
+17 10
+18 15
+19 5
+20 16
+21 9
+22 26
+23 9
+24 18
+25 23
+26 5
+27 17
+28 21
+29 26
+30 6
+31 22
+32 29
+33 23
+34 26
+35 33
+36 23
+37 11
+38 5
+39 26
+40 8
+41 6
+42 16
+43 14
+44 33
+45 9
+46 6
+DEPOT_SECTION
+1
+EOF
diff --git a/benchmarks/VehicleRouting/CVRP/data/instances/VRP-48-7.vrp b/benchmarks/VehicleRouting/CVRP/data/instances/VRP-48-7.vrp
new file mode 100644
index 00000000..92827102
--- /dev/null
+++ b/benchmarks/VehicleRouting/CVRP/data/instances/VRP-48-7.vrp
@@ -0,0 +1,108 @@
+NAME: VRP-48-7
+COMMENT: generated clustered CVRP instance (deterministic)
+TYPE: CVRP
+DIMENSION: 49
+CAPACITY: 170
+NODE_COORD_SECTION
+1 50.0 50.0
+2 21 64
+3 18 26
+4 49 31
+5 54 37
+6 31 69
+7 10 77
+8 2 10
+9 50 40
+10 54 40
+11 20 63
+12 27 88
+13 21 36
+14 46 37
+15 58 40
+16 44 78
+17 10 81
+18 9 34
+19 36 45
+20 58 45
+21 21 65
+22 28 95
+23 21 20
+24 41 52
+25 59 39
+26 34 54
+27 20 71
+28 20 21
+29 30 45
+30 49 50
+31 36 66
+32 31 78
+33 15 34
+34 50 37
+35 47 33
+36 12 68
+37 26 85
+38 19 25
+39 29 47
+40 55 38
+41 38 55
+42 23 81
+43 21 34
+44 43 48
+45 64 38
+46 40 49
+47 15 85
+48 19 31
+49 45 53
+DEMAND_SECTION
+1 0
+2 28
+3 18
+4 8
+5 9
+6 21
+7 33
+8 19
+9 33
+10 29
+11 28
+12 7
+13 25
+14 6
+15 5
+16 5
+17 25
+18 27
+19 6
+20 16
+21 16
+22 8
+23 5
+24 16
+25 25
+26 24
+27 29
+28 32
+29 14
+30 21
+31 17
+32 30
+33 10
+34 22
+35 5
+36 35
+37 30
+38 8
+39 29
+40 35
+41 5
+42 29
+43 5
+44 29
+45 20
+46 15
+47 35
+48 10
+49 30
+DEPOT_SECTION
+1
+EOF
diff --git a/benchmarks/VehicleRouting/CVRP/data/instances/VRP-54-8.vrp b/benchmarks/VehicleRouting/CVRP/data/instances/VRP-54-8.vrp
new file mode 100644
index 00000000..ddd6a6f6
--- /dev/null
+++ b/benchmarks/VehicleRouting/CVRP/data/instances/VRP-54-8.vrp
@@ -0,0 +1,120 @@
+NAME: VRP-54-8
+COMMENT: generated clustered CVRP instance (deterministic)
+TYPE: CVRP
+DIMENSION: 55
+CAPACITY: 175
+NODE_COORD_SECTION
+1 50.0 50.0
+2 27 67
+3 87 27
+4 48 19
+5 68 16
+6 63 21
+7 22 78
+8 40 73
+9 81 20
+10 55 44
+11 68 12
+12 66 27
+13 17 80
+14 51 69
+15 78 18
+16 50 39
+17 54 23
+18 66 37
+19 11 83
+20 24 60
+21 73 24
+22 51 30
+23 58 7
+24 77 34
+25 17 77
+26 35 62
+27 78 15
+28 65 21
+29 67 26
+30 65 30
+31 29 82
+32 38 46
+33 85 29
+34 64 39
+35 79 25
+36 56 26
+37 11 73
+38 37 71
+39 85 28
+40 40 30
+41 67 23
+42 69 38
+43 21 78
+44 33 68
+45 82 13
+46 49 39
+47 59 13
+48 72 30
+49 14 99
+50 41 62
+51 98 21
+52 62 30
+53 70 16
+54 63 21
+55 20 78
+DEMAND_SECTION
+1 0
+2 21
+3 21
+4 21
+5 28
+6 35
+7 11
+8 32
+9 10
+10 26
+11 33
+12 11
+13 14
+14 22
+15 8
+16 25
+17 28
+18 29
+19 6
+20 26
+21 26
+22 5
+23 5
+24 14
+25 9
+26 28
+27 9
+28 9
+29 33
+30 26
+31 10
+32 18
+33 35
+34 19
+35 28
+36 18
+37 12
+38 32
+39 32
+40 12
+41 16
+42 22
+43 24
+44 27
+45 11
+46 12
+47 33
+48 8
+49 23
+50 22
+51 27
+52 25
+53 23
+54 11
+55 34
+DEPOT_SECTION
+1
+EOF
diff --git a/benchmarks/VehicleRouting/CVRP/data/instances/VRP-55-8.vrp b/benchmarks/VehicleRouting/CVRP/data/instances/VRP-55-8.vrp
new file mode 100644
index 00000000..60b42757
--- /dev/null
+++ b/benchmarks/VehicleRouting/CVRP/data/instances/VRP-55-8.vrp
@@ -0,0 +1,122 @@
+NAME: VRP-55-8
+COMMENT: generated clustered CVRP instance (deterministic)
+TYPE: CVRP
+DIMENSION: 56
+CAPACITY: 185
+NODE_COORD_SECTION
+1 50.0 50.0
+2 95 56
+3 42 60
+4 40 63
+5 52 37
+6 80 37
+7 29 40
+8 84 49
+9 49 61
+10 36 64
+11 50 40
+12 82 36
+13 39 44
+14 94 39
+15 56 54
+16 43 60
+17 44 35
+18 83 38
+19 36 36
+20 90 43
+21 53 49
+22 33 72
+23 50 26
+24 76 33
+25 27 44
+26 72 41
+27 48 50
+28 31 70
+29 61 29
+30 52 42
+31 31 43
+32 93 46
+33 43 46
+34 24 72
+35 49 27
+36 73 39
+37 39 43
+38 68 49
+39 45 53
+40 45 58
+41 41 36
+42 71 34
+43 34 37
+44 75 49
+45 54 48
+46 41 68
+47 51 27
+48 69 53
+49 32 31
+50 77 38
+51 49 57
+52 44 65
+53 54 29
+54 91 39
+55 33 51
+56 88 45
+DEMAND_SECTION
+1 0
+2 31
+3 18
+4 34
+5 30
+6 12
+7 16
+8 30
+9 16
+10 13
+11 21
+12 24
+13 31
+14 29
+15 8
+16 21
+17 5
+18 35
+19 18
+20 35
+21 25
+22 32
+23 30
+24 24
+25 27
+26 31
+27 29
+28 31
+29 8
+30 9
+31 6
+32 16
+33 29
+34 20
+35 19
+36 17
+37 31
+38 18
+39 28
+40 20
+41 14
+42 12
+43 7
+44 26
+45 35
+46 31
+47 12
+48 26
+49 23
+50 16
+51 14
+52 24
+53 17
+54 7
+55 24
+56 5
+DEPOT_SECTION
+1
+EOF
diff --git a/benchmarks/VehicleRouting/CVRP/data/instances/VRP-60-10.vrp b/benchmarks/VehicleRouting/CVRP/data/instances/VRP-60-10.vrp
new file mode 100644
index 00000000..eee901e5
--- /dev/null
+++ b/benchmarks/VehicleRouting/CVRP/data/instances/VRP-60-10.vrp
@@ -0,0 +1,132 @@
+NAME: VRP-60-10
+COMMENT: generated clustered CVRP instance (deterministic)
+TYPE: CVRP
+DIMENSION: 61
+CAPACITY: 165
+NODE_COORD_SECTION
+1 50.0 50.0
+2 23 27
+3 70 33
+4 54 48
+5 70 60
+6 26 43
+7 47 72
+8 39 6
+9 84 29
+10 57 39
+11 75 62
+12 28 48
+13 52 62
+14 41 27
+15 81 33
+16 61 43
+17 74 58
+18 25 29
+19 40 68
+20 33 29
+21 79 28
+22 54 47
+23 70 55
+24 23 43
+25 50 65
+26 39 7
+27 74 38
+28 64 46
+29 75 66
+30 30 33
+31 71 66
+32 37 16
+33 84 27
+34 58 41
+35 65 57
+36 20 36
+37 58 66
+38 40 16
+39 81 37
+40 65 39
+41 84 65
+42 29 40
+43 56 74
+44 38 7
+45 64 32
+46 52 46
+47 66 57
+48 24 28
+49 60 76
+50 31 12
+51 70 41
+52 62 37
+53 59 46
+54 14 38
+55 59 62
+56 37 21
+57 73 32
+58 61 42
+59 57 58
+60 23 44
+61 56 68
+DEMAND_SECTION
+1 0
+2 15
+3 13
+4 11
+5 28
+6 14
+7 34
+8 34
+9 23
+10 9
+11 34
+12 19
+13 30
+14 9
+15 17
+16 18
+17 22
+18 27
+19 31
+20 20
+21 34
+22 25
+23 20
+24 9
+25 15
+26 18
+27 8
+28 18
+29 21
+30 32
+31 27
+32 20
+33 33
+34 33
+35 23
+36 14
+37 10
+38 25
+39 28
+40 11
+41 19
+42 34
+43 25
+44 19
+45 16
+46 24
+47 24
+48 23
+49 20
+50 24
+51 20
+52 35
+53 10
+54 20
+55 18
+56 33
+57 12
+58 33
+59 20
+60 32
+61 24
+DEPOT_SECTION
+1
+EOF
diff --git a/benchmarks/VehicleRouting/CVRP/data/instances/VRP-60-9.vrp b/benchmarks/VehicleRouting/CVRP/data/instances/VRP-60-9.vrp
new file mode 100644
index 00000000..39562dcb
--- /dev/null
+++ b/benchmarks/VehicleRouting/CVRP/data/instances/VRP-60-9.vrp
@@ -0,0 +1,132 @@
+NAME: VRP-60-9
+COMMENT: generated clustered CVRP instance (deterministic)
+TYPE: CVRP
+DIMENSION: 61
+CAPACITY: 160
+NODE_COORD_SECTION
+1 50.0 50.0
+2 75 51
+3 75 73
+4 52 27
+5 60 30
+6 37 40
+7 63 64
+8 86 66
+9 85 66
+10 63 23
+11 64 23
+12 36 41
+13 60 48
+14 63 59
+15 87 63
+16 63 10
+17 63 20
+18 28 38
+19 64 65
+20 69 48
+21 83 66
+22 71 18
+23 45 32
+24 42 60
+25 64 67
+26 72 54
+27 78 63
+28 67 18
+29 68 47
+30 40 52
+31 42 53
+32 70 46
+33 71 59
+34 65 29
+35 53 33
+36 26 41
+37 63 64
+38 76 57
+39 76 62
+40 65 25
+41 52 40
+42 42 31
+43 49 58
+44 67 54
+45 80 69
+46 66 31
+47 63 37
+48 43 45
+49 59 70
+50 66 49
+51 74 61
+52 60 13
+53 79 33
+54 43 43
+55 56 57
+56 56 54
+57 75 68
+58 57 13
+59 53 31
+60 29 45
+61 56 62
+DEMAND_SECTION
+1 0
+2 22
+3 6
+4 23
+5 25
+6 16
+7 8
+8 23
+9 8
+10 7
+11 17
+12 15
+13 23
+14 17
+15 13
+16 17
+17 5
+18 19
+19 12
+20 15
+21 18
+22 13
+23 33
+24 33
+25 27
+26 6
+27 28
+28 17
+29 28
+30 35
+31 15
+32 35
+33 16
+34 21
+35 9
+36 24
+37 24
+38 34
+39 5
+40 13
+41 18
+42 26
+43 17
+44 32
+45 27
+46 19
+47 19
+48 25
+49 15
+50 11
+51 13
+52 18
+53 29
+54 24
+55 8
+56 8
+57 7
+58 26
+59 25
+60 7
+61 35
+DEPOT_SECTION
+1
+EOF
diff --git a/benchmarks/VehicleRouting/CVRP/data/instances_heldout/VHO-22-3.vrp b/benchmarks/VehicleRouting/CVRP/data/instances_heldout/VHO-22-3.vrp
new file mode 100644
index 00000000..fba6e917
--- /dev/null
+++ b/benchmarks/VehicleRouting/CVRP/data/instances_heldout/VHO-22-3.vrp
@@ -0,0 +1,56 @@
+NAME: VHO-22-3
+COMMENT: generated clustered CVRP instance (deterministic)
+TYPE: CVRP
+DIMENSION: 23
+CAPACITY: 200
+NODE_COORD_SECTION
+1 50.0 50.0
+2 67 47
+3 40 68
+4 36 14
+5 68 64
+6 40 61
+7 29 27
+8 59 55
+9 42 42
+10 20 20
+11 67 46
+12 42 38
+13 21 11
+14 71 45
+15 43 48
+16 37 31
+17 67 41
+18 53 50
+19 40 27
+20 53 50
+21 52 51
+22 28 1
+23 59 66
+DEMAND_SECTION
+1 0
+2 26
+3 32
+4 17
+5 27
+6 9
+7 6
+8 25
+9 27
+10 17
+11 33
+12 17
+13 11
+14 30
+15 13
+16 10
+17 25
+18 33
+19 20
+20 24
+21 28
+22 35
+23 14
+DEPOT_SECTION
+1
+EOF
diff --git a/benchmarks/VehicleRouting/CVRP/data/instances_heldout/VHO-24-3.vrp b/benchmarks/VehicleRouting/CVRP/data/instances_heldout/VHO-24-3.vrp
new file mode 100644
index 00000000..e9a68d83
--- /dev/null
+++ b/benchmarks/VehicleRouting/CVRP/data/instances_heldout/VHO-24-3.vrp
@@ -0,0 +1,60 @@
+NAME: VHO-24-3
+COMMENT: generated clustered CVRP instance (deterministic)
+TYPE: CVRP
+DIMENSION: 25
+CAPACITY: 185
+NODE_COORD_SECTION
+1 50.0 50.0
+2 42 75
+3 48 11
+4 38 33
+5 26 68
+6 48 25
+7 51 31
+8 41 79
+9 50 29
+10 36 34
+11 48 56
+12 46 13
+13 36 29
+14 33 69
+15 60 13
+16 46 28
+17 28 87
+18 37 27
+19 55 31
+20 31 72
+21 43 21
+22 44 26
+23 39 81
+24 44 14
+25 43 29
+DEMAND_SECTION
+1 0
+2 12
+3 27
+4 20
+5 16
+6 5
+7 8
+8 25
+9 20
+10 11
+11 9
+12 11
+13 29
+14 33
+15 26
+16 30
+17 14
+18 22
+19 27
+20 7
+21 34
+22 9
+23 20
+24 17
+25 11
+DEPOT_SECTION
+1
+EOF
diff --git a/benchmarks/VehicleRouting/CVRP/data/instances_heldout/VHO-27-4.vrp b/benchmarks/VehicleRouting/CVRP/data/instances_heldout/VHO-27-4.vrp
new file mode 100644
index 00000000..6c705691
--- /dev/null
+++ b/benchmarks/VehicleRouting/CVRP/data/instances_heldout/VHO-27-4.vrp
@@ -0,0 +1,66 @@
+NAME: VHO-27-4
+COMMENT: generated clustered CVRP instance (deterministic)
+TYPE: CVRP
+DIMENSION: 28
+CAPACITY: 200
+NODE_COORD_SECTION
+1 50.0 50.0
+2 66 42
+3 41 7
+4 30 52
+5 53 42
+6 30 27
+7 51 55
+8 53 38
+9 39 20
+10 34 51
+11 43 33
+12 32 19
+13 51 63
+14 53 50
+15 46 22
+16 41 55
+17 62 46
+18 44 24
+19 30 56
+20 42 49
+21 47 14
+22 37 59
+23 60 51
+24 41 13
+25 35 66
+26 47 53
+27 24 11
+28 29 64
+DEMAND_SECTION
+1 0
+2 31
+3 16
+4 21
+5 7
+6 21
+7 34
+8 32
+9 24
+10 33
+11 22
+12 22
+13 13
+14 35
+15 28
+16 6
+17 34
+18 18
+19 32
+20 8
+21 15
+22 35
+23 31
+24 26
+25 11
+26 28
+27 25
+28 23
+DEPOT_SECTION
+1
+EOF
diff --git a/benchmarks/VehicleRouting/CVRP/data/instances_heldout/VHO-29-4.vrp b/benchmarks/VehicleRouting/CVRP/data/instances_heldout/VHO-29-4.vrp
new file mode 100644
index 00000000..b7a39792
--- /dev/null
+++ b/benchmarks/VehicleRouting/CVRP/data/instances_heldout/VHO-29-4.vrp
@@ -0,0 +1,70 @@
+NAME: VHO-29-4
+COMMENT: generated clustered CVRP instance (deterministic)
+TYPE: CVRP
+DIMENSION: 30
+CAPACITY: 195
+NODE_COORD_SECTION
+1 50.0 50.0
+2 35 83
+3 64 31
+4 74 23
+5 16 2
+6 37 76
+7 47 31
+8 65 43
+9 30 13
+10 29 81
+11 49 30
+12 55 43
+13 28 13
+14 45 76
+15 57 19
+16 67 43
+17 25 24
+18 50 71
+19 59 23
+20 72 41
+21 21 28
+22 27 85
+23 49 37
+24 64 44
+25 8 13
+26 38 82
+27 54 38
+28 59 37
+29 19 23
+30 48 88
+DEMAND_SECTION
+1 0
+2 10
+3 27
+4 23
+5 17
+6 13
+7 23
+8 33
+9 29
+10 24
+11 27
+12 12
+13 15
+14 23
+15 22
+16 12
+17 26
+18 7
+19 34
+20 35
+21 16
+22 26
+23 27
+24 32
+25 5
+26 34
+27 15
+28 10
+29 15
+30 24
+DEPOT_SECTION
+1
+EOF
diff --git a/benchmarks/VehicleRouting/CVRP/data/instances_heldout/VHO-33-5.vrp b/benchmarks/VehicleRouting/CVRP/data/instances_heldout/VHO-33-5.vrp
new file mode 100644
index 00000000..3ccc8649
--- /dev/null
+++ b/benchmarks/VehicleRouting/CVRP/data/instances_heldout/VHO-33-5.vrp
@@ -0,0 +1,78 @@
+NAME: VHO-33-5
+COMMENT: generated clustered CVRP instance (deterministic)
+TYPE: CVRP
+DIMENSION: 34
+CAPACITY: 170
+NODE_COORD_SECTION
+1 50.0 50.0
+2 21 19
+3 55 60
+4 81 28
+5 65 76
+6 20 34
+7 46 81
+8 79 22
+9 72 70
+10 17 55
+11 65 69
+12 74 21
+13 73 73
+14 31 42
+15 54 75
+16 71 13
+17 58 86
+18 20 39
+19 61 61
+20 87 20
+21 67 74
+22 20 35
+23 51 75
+24 76 21
+25 65 73
+26 31 27
+27 64 66
+28 83 22
+29 62 76
+30 16 34
+31 49 74
+32 85 17
+33 69 66
+34 8 32
+DEMAND_SECTION
+1 0
+2 32
+3 33
+4 30
+5 28
+6 8
+7 23
+8 19
+9 5
+10 12
+11 34
+12 34
+13 7
+14 10
+15 34
+16 19
+17 27
+18 7
+19 20
+20 17
+21 17
+22 6
+23 35
+24 25
+25 17
+26 7
+27 32
+28 23
+29 13
+30 26
+31 12
+32 16
+33 29
+34 10
+DEPOT_SECTION
+1
+EOF
diff --git a/benchmarks/VehicleRouting/CVRP/data/instances_heldout/VHO-36-5.vrp b/benchmarks/VehicleRouting/CVRP/data/instances_heldout/VHO-36-5.vrp
new file mode 100644
index 00000000..790921e0
--- /dev/null
+++ b/benchmarks/VehicleRouting/CVRP/data/instances_heldout/VHO-36-5.vrp
@@ -0,0 +1,84 @@
+NAME: VHO-36-5
+COMMENT: generated clustered CVRP instance (deterministic)
+TYPE: CVRP
+DIMENSION: 37
+CAPACITY: 185
+NODE_COORD_SECTION
+1 50.0 50.0
+2 77 80
+3 51 33
+4 42 31
+5 40 81
+6 50 76
+7 45 35
+8 65 33
+9 34 72
+10 63 56
+11 42 53
+12 56 45
+13 24 73
+14 73 69
+15 55 37
+16 58 37
+17 40 72
+18 65 70
+19 49 37
+20 55 49
+21 35 82
+22 62 59
+23 37 42
+24 47 50
+25 36 85
+26 72 66
+27 44 31
+28 52 33
+29 19 71
+30 72 62
+31 46 52
+32 54 25
+33 33 81
+34 67 75
+35 48 46
+36 50 42
+37 38 67
+DEMAND_SECTION
+1 0
+2 18
+3 8
+4 8
+5 21
+6 25
+7 28
+8 11
+9 28
+10 5
+11 24
+12 32
+13 15
+14 11
+15 16
+16 16
+17 24
+18 27
+19 33
+20 20
+21 8
+22 22
+23 20
+24 9
+25 18
+26 33
+27 30
+28 10
+29 27
+30 30
+31 18
+32 26
+33 33
+34 17
+35 7
+36 12
+37 31
+DEPOT_SECTION
+1
+EOF
diff --git a/benchmarks/VehicleRouting/CVRP/data/instances_heldout/VHO-39-6.vrp b/benchmarks/VehicleRouting/CVRP/data/instances_heldout/VHO-39-6.vrp
new file mode 100644
index 00000000..4df962a6
--- /dev/null
+++ b/benchmarks/VehicleRouting/CVRP/data/instances_heldout/VHO-39-6.vrp
@@ -0,0 +1,90 @@
+NAME: VHO-39-6
+COMMENT: generated clustered CVRP instance (deterministic)
+TYPE: CVRP
+DIMENSION: 40
+CAPACITY: 160
+NODE_COORD_SECTION
+1 50.0 50.0
+2 59 27
+3 58 27
+4 83 72
+5 74 56
+6 31 46
+7 55 32
+8 46 31
+9 87 56
+10 82 46
+11 36 46
+12 67 10
+13 46 25
+14 80 64
+15 78 54
+16 28 65
+17 63 35
+18 59 26
+19 91 61
+20 81 53
+21 43 48
+22 72 33
+23 47 34
+24 88 61
+25 89 53
+26 40 59
+27 53 32
+28 50 26
+29 100 63
+30 72 49
+31 29 57
+32 70 26
+33 53 37
+34 78 45
+35 76 69
+36 43 40
+37 72 36
+38 58 28
+39 73 80
+40 74 41
+DEMAND_SECTION
+1 0
+2 21
+3 29
+4 20
+5 28
+6 15
+7 35
+8 30
+9 16
+10 6
+11 22
+12 17
+13 5
+14 12
+15 21
+16 7
+17 34
+18 23
+19 10
+20 11
+21 8
+22 32
+23 9
+24 15
+25 19
+26 21
+27 20
+28 22
+29 7
+30 31
+31 23
+32 34
+33 15
+34 9
+35 13
+36 32
+37 5
+38 28
+39 26
+40 30
+DEPOT_SECTION
+1
+EOF
diff --git a/benchmarks/VehicleRouting/CVRP/data/instances_heldout/VHO-43-6.vrp b/benchmarks/VehicleRouting/CVRP/data/instances_heldout/VHO-43-6.vrp
new file mode 100644
index 00000000..edf7d8ab
--- /dev/null
+++ b/benchmarks/VehicleRouting/CVRP/data/instances_heldout/VHO-43-6.vrp
@@ -0,0 +1,98 @@
+NAME: VHO-43-6
+COMMENT: generated clustered CVRP instance (deterministic)
+TYPE: CVRP
+DIMENSION: 44
+CAPACITY: 180
+NODE_COORD_SECTION
+1 50.0 50.0
+2 41 47
+3 30 40
+4 89 46
+5 51 41
+6 79 25
+7 27 50
+8 21 32
+9 76 37
+10 47 26
+11 72 36
+12 27 56
+13 20 34
+14 78 36
+15 62 44
+16 59 32
+17 24 74
+18 20 42
+19 86 28
+20 41 42
+21 83 25
+22 35 59
+23 18 45
+24 78 29
+25 47 54
+26 78 33
+27 37 72
+28 18 36
+29 89 54
+30 36 54
+31 69 32
+32 39 64
+33 24 33
+34 77 42
+35 35 46
+36 78 40
+37 35 60
+38 13 40
+39 82 42
+40 49 45
+41 72 32
+42 42 43
+43 25 45
+44 91 35
+DEMAND_SECTION
+1 0
+2 26
+3 16
+4 25
+5 22
+6 16
+7 10
+8 33
+9 12
+10 7
+11 14
+12 27
+13 14
+14 27
+15 22
+16 18
+17 9
+18 35
+19 15
+20 15
+21 16
+22 24
+23 21
+24 31
+25 16
+26 21
+27 30
+28 16
+29 18
+30 30
+31 20
+32 23
+33 29
+34 15
+35 34
+36 35
+37 14
+38 6
+39 8
+40 8
+41 10
+42 6
+43 30
+44 26
+DEPOT_SECTION
+1
+EOF
diff --git a/benchmarks/VehicleRouting/CVRP/data/instances_heldout/VHO-47-7.vrp b/benchmarks/VehicleRouting/CVRP/data/instances_heldout/VHO-47-7.vrp
new file mode 100644
index 00000000..f511ef8c
--- /dev/null
+++ b/benchmarks/VehicleRouting/CVRP/data/instances_heldout/VHO-47-7.vrp
@@ -0,0 +1,106 @@
+NAME: VHO-47-7
+COMMENT: generated clustered CVRP instance (deterministic)
+TYPE: CVRP
+DIMENSION: 48
+CAPACITY: 165
+NODE_COORD_SECTION
+1 50.0 50.0
+2 48 78
+3 64 64
+4 59 50
+5 77 74
+6 30 80
+7 53 79
+8 76 69
+9 67 43
+10 76 70
+11 27 83
+12 52 90
+13 65 66
+14 70 29
+15 71 67
+16 35 84
+17 59 65
+18 66 70
+19 68 28
+20 82 75
+21 32 69
+22 57 82
+23 70 64
+24 69 22
+25 91 76
+26 16 87
+27 33 81
+28 56 53
+29 77 45
+30 75 71
+31 19 92
+32 53 62
+33 76 56
+34 68 27
+35 76 73
+36 36 85
+37 50 76
+38 66 66
+39 49 30
+40 79 88
+41 33 77
+42 40 70
+43 73 59
+44 72 41
+45 68 84
+46 27 72
+47 43 74
+48 80 76
+DEMAND_SECTION
+1 0
+2 34
+3 32
+4 25
+5 9
+6 12
+7 6
+8 27
+9 16
+10 22
+11 23
+12 14
+13 14
+14 22
+15 29
+16 34
+17 8
+18 24
+19 29
+20 26
+21 18
+22 16
+23 9
+24 17
+25 22
+26 16
+27 29
+28 20
+29 26
+30 6
+31 21
+32 27
+33 24
+34 31
+35 35
+36 34
+37 30
+38 12
+39 11
+40 5
+41 5
+42 5
+43 18
+44 9
+45 21
+46 10
+47 31
+48 9
+DEPOT_SECTION
+1
+EOF
diff --git a/benchmarks/VehicleRouting/CVRP/data/instances_heldout/VHO-51-7.vrp b/benchmarks/VehicleRouting/CVRP/data/instances_heldout/VHO-51-7.vrp
new file mode 100644
index 00000000..57ddc3c8
--- /dev/null
+++ b/benchmarks/VehicleRouting/CVRP/data/instances_heldout/VHO-51-7.vrp
@@ -0,0 +1,114 @@
+NAME: VHO-51-7
+COMMENT: generated clustered CVRP instance (deterministic)
+TYPE: CVRP
+DIMENSION: 52
+CAPACITY: 195
+NODE_COORD_SECTION
+1 50.0 50.0
+2 53 46
+3 54 55
+4 46 87
+5 31 27
+6 74 54
+7 44 54
+8 46 37
+9 65 63
+10 51 91
+11 30 34
+12 81 45
+13 49 45
+14 49 41
+15 57 57
+16 39 84
+17 43 13
+18 65 66
+19 44 57
+20 43 40
+21 64 53
+22 35 78
+23 40 26
+24 81 42
+25 51 44
+26 41 42
+27 52 41
+28 53 81
+29 48 20
+30 79 47
+31 56 57
+32 31 45
+33 65 64
+34 44 65
+35 35 33
+36 85 44
+37 45 52
+38 47 43
+39 60 62
+40 43 85
+41 44 20
+42 75 47
+43 52 54
+44 38 41
+45 60 55
+46 45 85
+47 34 30
+48 82 46
+49 53 39
+50 47 46
+51 46 57
+52 45 80
+DEMAND_SECTION
+1 0
+2 16
+3 29
+4 20
+5 21
+6 20
+7 21
+8 23
+9 21
+10 26
+11 30
+12 7
+13 7
+14 21
+15 34
+16 18
+17 18
+18 11
+19 31
+20 24
+21 13
+22 5
+23 32
+24 13
+25 18
+26 17
+27 28
+28 23
+29 34
+30 6
+31 25
+32 17
+33 15
+34 34
+35 33
+36 29
+37 5
+38 16
+39 7
+40 12
+41 19
+42 12
+43 16
+44 31
+45 10
+46 29
+47 27
+48 35
+49 21
+50 25
+51 32
+52 31
+DEPOT_SECTION
+1
+EOF
diff --git a/benchmarks/VehicleRouting/CVRP/data/instances_heldout/VHO-55-8.vrp b/benchmarks/VehicleRouting/CVRP/data/instances_heldout/VHO-55-8.vrp
new file mode 100644
index 00000000..78f1e4aa
--- /dev/null
+++ b/benchmarks/VehicleRouting/CVRP/data/instances_heldout/VHO-55-8.vrp
@@ -0,0 +1,122 @@
+NAME: VHO-55-8
+COMMENT: generated clustered CVRP instance (deterministic)
+TYPE: CVRP
+DIMENSION: 56
+CAPACITY: 185
+NODE_COORD_SECTION
+1 50.0 50.0
+2 15 30
+3 89 20
+4 24 41
+5 48 21
+6 28 55
+7 69 24
+8 33 19
+9 90 38
+10 32 36
+11 41 22
+12 29 61
+13 66 24
+14 14 11
+15 89 22
+16 25 37
+17 45 36
+18 17 61
+19 53 28
+20 12 27
+21 80 37
+22 25 38
+23 49 22
+24 29 55
+25 53 41
+26 30 15
+27 86 23
+28 27 41
+29 36 42
+30 22 61
+31 61 33
+32 18 16
+33 84 20
+34 23 45
+35 55 26
+36 19 68
+37 62 28
+38 17 4
+39 93 31
+40 29 35
+41 48 32
+42 20 68
+43 59 23
+44 7 20
+45 92 27
+46 28 32
+47 58 43
+48 38 59
+49 58 16
+50 7 26
+51 87 28
+52 23 33
+53 50 39
+54 30 63
+55 51 16
+56 13 7
+DEMAND_SECTION
+1 0
+2 20
+3 32
+4 18
+5 28
+6 24
+7 24
+8 16
+9 27
+10 5
+11 6
+12 18
+13 29
+14 14
+15 5
+16 16
+17 33
+18 29
+19 24
+20 19
+21 14
+22 7
+23 34
+24 18
+25 17
+26 33
+27 22
+28 27
+29 31
+30 35
+31 10
+32 20
+33 29
+34 8
+35 21
+36 32
+37 24
+38 25
+39 17
+40 21
+41 11
+42 29
+43 17
+44 6
+45 15
+46 14
+47 26
+48 23
+49 35
+50 19
+51 18
+52 29
+53 25
+54 30
+55 32
+56 20
+DEPOT_SECTION
+1
+EOF
diff --git a/benchmarks/VehicleRouting/CVRP/data/instances_heldout/VHO-58-9.vrp b/benchmarks/VehicleRouting/CVRP/data/instances_heldout/VHO-58-9.vrp
new file mode 100644
index 00000000..2515cc4d
--- /dev/null
+++ b/benchmarks/VehicleRouting/CVRP/data/instances_heldout/VHO-58-9.vrp
@@ -0,0 +1,128 @@
+NAME: VHO-58-9
+COMMENT: generated clustered CVRP instance (deterministic)
+TYPE: CVRP
+DIMENSION: 59
+CAPACITY: 165
+NODE_COORD_SECTION
+1 50.0 50.0
+2 79 72
+3 43 44
+4 39 54
+5 36 42
+6 22 29
+7 85 76
+8 69 68
+9 31 43
+10 30 65
+11 45 41
+12 35 28
+13 63 70
+14 70 68
+15 33 37
+16 32 72
+17 45 40
+18 32 25
+19 79 79
+20 70 71
+21 41 43
+22 39 68
+23 43 34
+24 26 31
+25 78 88
+26 62 56
+27 41 60
+28 31 70
+29 44 60
+30 22 18
+31 78 78
+32 69 61
+33 43 39
+34 22 64
+35 42 39
+36 22 30
+37 85 84
+38 75 65
+39 52 49
+40 32 60
+41 42 45
+42 27 21
+43 85 86
+44 64 70
+45 34 51
+46 32 70
+47 49 39
+48 25 16
+49 79 87
+50 80 62
+51 40 58
+52 27 55
+53 47 36
+54 30 13
+55 81 67
+56 72 70
+57 50 45
+58 34 69
+59 32 43
+DEMAND_SECTION
+1 0
+2 34
+3 21
+4 8
+5 27
+6 33
+7 18
+8 13
+9 10
+10 28
+11 24
+12 23
+13 10
+14 18
+15 20
+16 29
+17 34
+18 18
+19 13
+20 33
+21 28
+22 31
+23 34
+24 27
+25 24
+26 31
+27 13
+28 29
+29 5
+30 30
+31 19
+32 20
+33 25
+34 16
+35 29
+36 16
+37 7
+38 32
+39 31
+40 27
+41 25
+42 17
+43 32
+44 17
+45 7
+46 15
+47 9
+48 19
+49 11
+50 13
+51 20
+52 10
+53 7
+54 12
+55 11
+56 7
+57 24
+58 9
+59 30
+DEPOT_SECTION
+1
+EOF
diff --git a/benchmarks/VehicleRouting/CVRP/data/reference.json b/benchmarks/VehicleRouting/CVRP/data/reference.json
new file mode 100644
index 00000000..35974de1
--- /dev/null
+++ b/benchmarks/VehicleRouting/CVRP/data/reference.json
@@ -0,0 +1,26 @@
+{
+ "VHO-22-3": 238,
+ "VHO-24-3": 256,
+ "VHO-27-4": 279,
+ "VHO-29-4": 379,
+ "VHO-33-5": 476,
+ "VHO-36-5": 355,
+ "VHO-39-6": 469,
+ "VHO-43-6": 452,
+ "VHO-47-7": 562,
+ "VHO-51-7": 424,
+ "VHO-55-8": 648,
+ "VHO-58-9": 576,
+ "VRP-19-2": 153,
+ "VRP-21-3": 254,
+ "VRP-22-4": 313,
+ "VRP-32-5": 394,
+ "VRP-37-6": 479,
+ "VRP-45-6": 431,
+ "VRP-45-7": 569,
+ "VRP-48-7": 530,
+ "VRP-54-8": 672,
+ "VRP-55-8": 480,
+ "VRP-60-10": 649,
+ "VRP-60-9": 527
+}
diff --git a/benchmarks/VehicleRouting/CVRP/frontier_eval/agent_files.txt b/benchmarks/VehicleRouting/CVRP/frontier_eval/agent_files.txt
new file mode 100644
index 00000000..640607a1
--- /dev/null
+++ b/benchmarks/VehicleRouting/CVRP/frontier_eval/agent_files.txt
@@ -0,0 +1,4 @@
+README.md
+Task.md
+baseline/solver.py
+frontier_eval/constraints.txt
diff --git a/benchmarks/VehicleRouting/CVRP/frontier_eval/artifact_files.txt b/benchmarks/VehicleRouting/CVRP/frontier_eval/artifact_files.txt
new file mode 100644
index 00000000..a52b8b16
--- /dev/null
+++ b/benchmarks/VehicleRouting/CVRP/frontier_eval/artifact_files.txt
@@ -0,0 +1,2 @@
+# No extra artifact files are auto-collected by default for this benchmark.
+# metrics.json and artifacts.json are handled separately by UnifiedTask.
diff --git a/benchmarks/VehicleRouting/CVRP/frontier_eval/candidate_destination.txt b/benchmarks/VehicleRouting/CVRP/frontier_eval/candidate_destination.txt
new file mode 100644
index 00000000..6645b02f
--- /dev/null
+++ b/benchmarks/VehicleRouting/CVRP/frontier_eval/candidate_destination.txt
@@ -0,0 +1 @@
+baseline/solver.py
diff --git a/benchmarks/VehicleRouting/CVRP/frontier_eval/constraints.txt b/benchmarks/VehicleRouting/CVRP/frontier_eval/constraints.txt
new file mode 100644
index 00000000..c97f6a62
--- /dev/null
+++ b/benchmarks/VehicleRouting/CVRP/frontier_eval/constraints.txt
@@ -0,0 +1,9 @@
+UnifiedTask constraints:
+1) Only modify `baseline/solver.py`, and only inside the EVOLVE-BLOCK-START / EVOLVE-BLOCK-END region.
+2) Preserve the public entrypoint `solve(instance)` and its output contract (a list of routes; each route is a list of customer ids 1..n, depot implicit at both ends).
+3) Do not modify benchmark assets, documentation, verification code, runtime helpers, tests, data files, or `frontier_eval/` metadata.
+4) Never read `data/reference.json` and never import or call the reference solver (`verification/ref_solver.py`); both are the scoring baseline and are off-limits.
+5) Do not hardcode solutions per instance name: evaluation includes held-out instances you have never seen, so your solver must be a general algorithm.
+6) Your solver must always produce a valid solution: every customer served exactly once, every route within capacity, no crashing, no timeouts. Invalid output scores 0 and marks the run invalid.
+7) Prioritize validity and correctness before optimization.
+8) The evaluator statically checks violations of 1/4/5 (markers, forbidden references, absolute paths, per-instance hardcoding) and runs a determinism probe (the candidate must produce identical output on the same instance across two runs); any violation scores 0.
diff --git a/benchmarks/VehicleRouting/CVRP/frontier_eval/copy_files.txt b/benchmarks/VehicleRouting/CVRP/frontier_eval/copy_files.txt
new file mode 100644
index 00000000..b178885b
--- /dev/null
+++ b/benchmarks/VehicleRouting/CVRP/frontier_eval/copy_files.txt
@@ -0,0 +1,3 @@
+baseline
+data/instances
+frontier_eval
diff --git a/benchmarks/VehicleRouting/CVRP/frontier_eval/eval_command.txt b/benchmarks/VehicleRouting/CVRP/frontier_eval/eval_command.txt
new file mode 100644
index 00000000..3a31d525
--- /dev/null
+++ b/benchmarks/VehicleRouting/CVRP/frontier_eval/eval_command.txt
@@ -0,0 +1 @@
+FRONTIER_EVAL_UNIFIED_SOURCE_BENCHMARK_DIR={benchmark_source} {python} frontier_eval/run_eval.py --candidate {candidate} --metrics-out metrics.json --artifacts-out artifacts.json
diff --git a/benchmarks/VehicleRouting/CVRP/frontier_eval/eval_cwd.txt b/benchmarks/VehicleRouting/CVRP/frontier_eval/eval_cwd.txt
new file mode 100644
index 00000000..9c558e35
--- /dev/null
+++ b/benchmarks/VehicleRouting/CVRP/frontier_eval/eval_cwd.txt
@@ -0,0 +1 @@
+.
diff --git a/benchmarks/VehicleRouting/CVRP/frontier_eval/evaluator.py b/benchmarks/VehicleRouting/CVRP/frontier_eval/evaluator.py
new file mode 100644
index 00000000..66cce353
--- /dev/null
+++ b/benchmarks/VehicleRouting/CVRP/frontier_eval/evaluator.py
@@ -0,0 +1,539 @@
+"""Self-contained unified evaluator for the CVRP benchmark.
+
+This module runs entirely inside the evaluation sandbox. It embeds instance
+parsing, route validation, scoring and candidate-integrity checks, so no
+`verification/` files (including the reference solver) are copied into the
+sandbox.
+
+Security / integrity:
+ * Reference distances are read from the host benchmark directory
+ (FRONTIER_EVAL_UNIFIED_SOURCE_BENCHMARK_DIR) and are never copied into
+ the sandbox; the candidate subprocess runs without that variable and
+ without any reference-distance settings.
+ * The candidate is checked before running: EVOLVE-BLOCK markers must be
+ present, the fixed regions must match the initial baseline, and the
+ source must not reference the verification module / reference solver /
+ reference.json / absolute paths / per-instance hardcoded routes.
+ * A determinism probe runs the candidate twice on the smallest instance;
+ differing outputs invalidate the run.
+
+The reference implementation (`verification/ref_solver.py`) stays out of the
+sandbox and is only used to produce `data/reference.json` on the host.
+"""
+from __future__ import annotations
+
+import json
+import math
+import os
+import re
+import statistics
+import subprocess
+import sys
+import tempfile
+from pathlib import Path
+from typing import Any
+
+TASK_ROOT = Path(__file__).resolve().parents[1] # or CVRP dir
+INSTANCES_DIR = TASK_ROOT / "data" / "instances"
+HELDOUT_DIR = TASK_ROOT / "data" / "instances_heldout"
+REFERENCE_JSON = TASK_ROOT / "data" / "reference.json"
+
+EVOLVE_START = "# EVOLVE-BLOCK-START"
+EVOLVE_END = "# EVOLVE-BLOCK-END"
+
+# Strong tokens: almost never appear in legitimate solver code, so a bare
+# substring match is fine (e.g. "ref_solver", "reference.json").
+STRONG_TOKENS = (
+ "ref_solver",
+ "grasp_solve",
+ "savings_solve",
+ "reference.json",
+)
+# Weak token "verification": only rejected in an import/from/module-path
+# context, so a comment like "verification pass" is NOT a false positive.
+FORBIDDEN_RE = (
+ re.compile(r"\b(?:import|from)\s+verification\b"),
+ re.compile(r"verification[\\/.]"),
+)
+
+HARDCODE_RE = re.compile(r"[\"'][A-Z][A-Z0-9-]*\d+-\d+[\"']\s*:")
+ABS_PATH_RE = re.compile(r"[A-Za-z]:[\\/]|/home/|/Users/")
+
+DEFAULT_TIMEOUT_S = 60
+
+
+def parse_instance(path: Path) -> dict:
+ """Parse a TSPLIB-style CVRP .vrp file into an instance dict."""
+ text = path.read_text(encoding="utf-8", errors="ignore")
+ coords: dict[int, tuple[float, float]] = {}
+ demands: dict[int, int] = {}
+ capacity = 0
+ section = None
+ for line in text.splitlines():
+ line = line.strip()
+ if not line:
+ continue
+ upper = line.upper()
+ if upper.startswith("CAPACITY"):
+ capacity = int(line.split(":")[-1].strip())
+ continue
+ if upper == "NODE_COORD_SECTION":
+ section = "coords"
+ continue
+ if upper == "DEMAND_SECTION":
+ section = "demand"
+ continue
+ if upper == "DEPOT_SECTION":
+ section = None
+ continue
+ if upper == "EOF" or upper.startswith(("EDGE_WEIGHT", "DISPLAY_DATA")):
+ section = None
+ continue
+ if upper.startswith(("NAME", "COMMENT", "TYPE", "DIMENSION")):
+ continue
+ if section == "coords":
+ parts = line.split()
+ if len(parts) >= 3:
+ coords[int(parts[0])] = (float(parts[1]), float(parts[2]))
+ elif section == "demand":
+ parts = line.split()
+ if len(parts) >= 2:
+ demands[int(parts[0])] = int(parts[1])
+ n_customers = max(coords) - 1 # depot is id 1, customers are ids 2..n+1
+ pts = [coords[1]] + [coords[i] for i in range(2, n_customers + 2)]
+ dist = [[0] * (n_customers + 1) for _ in range(n_customers + 1)]
+ for i in range(n_customers + 1):
+ for j in range(n_customers + 1):
+ dx = pts[i][0] - pts[j][0]
+ dy = pts[i][1] - pts[j][1]
+ dist[i][j] = int(round(math.hypot(dx, dy)))
+ return {
+ "name": path.stem,
+ "n": n_customers,
+ "capacity": capacity,
+ "demand": [0] + [demands.get(i, 0) for i in range(2, n_customers + 2)],
+ "distance": dist,
+ }
+
+
+def route_distance(routes: list, dist: list[list[int]]) -> int:
+ total = 0
+ for seq in routes:
+ if not seq:
+ continue
+ total += dist[0][seq[0]]
+ for a, b in zip(seq, seq[1:]):
+ total += dist[a][b]
+ total += dist[seq[-1]][0]
+ return total
+
+
+def validate(routes: Any, inst: dict) -> tuple[bool, str, int | None]:
+ """Return (ok, error_message, total_distance)."""
+ n, cap, demand = inst["n"], inst["capacity"], inst["demand"]
+ if not isinstance(routes, list):
+ return False, "routes is not a list", None
+ seen: list[int] = []
+ for seq in routes:
+ if not isinstance(seq, list):
+ return False, "a route is not a list", None
+ load = 0
+ for c in seq:
+ if not isinstance(c, int) or isinstance(c, bool):
+ return False, f"route contains non-integer {c!r}", None
+ if c < 1 or c > n:
+ return False, f"customer id {c} out of range 1..{n}", None
+ if c in seen:
+ return False, f"customer {c} visited more than once", None
+ seen.append(c)
+ load += demand[c]
+ if load > cap:
+ return False, f"route {seq} exceeds capacity {cap} (load {load})", None
+ if set(seen) != set(range(1, n + 1)):
+ missing = set(range(1, n + 1)) - set(seen)
+ return False, f"customers not served: {sorted(missing)}", None
+ return True, "", route_distance(routes, inst["distance"])
+
+
+# ---------------------------------------------------------------------------
+# Candidate integrity checks (mirrors verification/validator.py so the sandbox
+# needs no verification/ files).
+# ---------------------------------------------------------------------------
+def split_evolve_blocks(src: str) -> tuple[str, str, str] | None:
+ start = src.find(EVOLVE_START)
+ end = src.find(EVOLVE_END)
+ if start == -1 or end == -1 or end <= start:
+ return None
+ return (
+ src[:start],
+ src[start + len(EVOLVE_START) : end],
+ src[end + len(EVOLVE_END) :],
+ )
+
+
+def fixed_region(parts: tuple[str, str, str]) -> str:
+ return parts[0] + parts[2]
+
+
+def static_check_source(src: str, baseline_src: str | None = None) -> list[str]:
+ issues: list[str] = []
+ parts = split_evolve_blocks(src)
+ if parts is None:
+ issues.append("missing EVOLVE-BLOCK-START / EVOLVE-BLOCK-END markers")
+ elif baseline_src is not None:
+ init_parts = split_evolve_blocks(baseline_src)
+ if init_parts is not None and fixed_region(init_parts) != fixed_region(parts):
+ issues.append("code outside EVOLVE-BLOCK differs from initial baseline")
+ for token in STRONG_TOKENS:
+ if token in src:
+ issues.append(f"candidate references forbidden token {token!r}")
+ for pat in FORBIDDEN_RE:
+ if pat.search(src):
+ issues.append("candidate references forbidden token 'verification'")
+ if ABS_PATH_RE.search(src):
+ issues.append("candidate contains an absolute filesystem path")
+ if HARDCODE_RE.search(src):
+ issues.append("candidate hardcodes per-instance solutions by name")
+ return issues
+
+
+def candidate_env() -> dict[str, str]:
+ """Candidate subprocess env, stripped of all `FRONTIER_*` (the unified
+ runtime sets `FRONTIER_ENGINEERING_ROOT` to the repo root — a candidate
+ could use it to find and import `verification/ref_solver.py` on the host)
+ and of reference-distance / generation-seed settings."""
+ env = os.environ.copy()
+ for key in list(env):
+ upper = key.upper()
+ if upper.startswith("FRONTIER") or key in (
+ "CVRP_EVAL_REFERENCE_JSON",
+ "CVRP_EVAL_REFERENCES",
+ "CVRP_EVAL_GENERATE_SEED",
+ "CVRP_EVAL_GENERATE_COUNT",
+ ):
+ del env[key]
+ return env
+
+
+def check_determinism(
+ python: str,
+ solver_path: Path,
+ inst_path: Path,
+ timeout: float,
+) -> tuple[bool, str]:
+ outputs: list[Any] = []
+ for _ in range(2):
+ with tempfile.TemporaryDirectory(prefix="cvrp_det_") as td:
+ out_path = Path(td) / "out.json"
+ try:
+ proc = subprocess.run(
+ [python, str(solver_path), str(inst_path), str(out_path)],
+ capture_output=True,
+ text=True,
+ timeout=timeout,
+ cwd=str(solver_path.parent),
+ env=candidate_env(),
+ )
+ except subprocess.TimeoutExpired:
+ return False, "timeout during determinism check"
+ if proc.returncode != 0:
+ return (
+ False,
+ f"candidate exited with code {proc.returncode} during "
+ f"determinism check: {(proc.stderr or '')[:200]}",
+ )
+ try:
+ outputs.append(json.loads(out_path.read_text(encoding="utf-8")))
+ except Exception as exc:
+ return False, f"cannot parse determinism output: {exc}"
+ if outputs[0] != outputs[1]:
+ return False, "candidate is not deterministic (output differs across two runs)"
+ return True, ""
+
+
+def select_determinism_probes(
+ inst_paths: list, parse_instance, count: int = 3
+) -> list:
+ """Pick min / median / max instances (by customer count) as determinism
+ probes, so a solver that is deterministic on small instances but random on
+ large ones cannot slip through a single-probe check."""
+ parsed = sorted(
+ ((p, parse_instance(p)) for p in inst_paths), key=lambda t: t[1]["n"]
+ )
+ if not parsed:
+ return []
+ idxs = sorted({0, len(parsed) // 2, len(parsed) - 1})
+ return [parsed[i][0] for i in idxs[:count]]
+
+
+# ---------------------------------------------------------------------------
+# Host-only access (reference distances + initial baseline live on the host).
+# ---------------------------------------------------------------------------
+def _source_benchmark_dir() -> Path | None:
+ raw = os.environ.get("FRONTIER_EVAL_UNIFIED_SOURCE_BENCHMARK_DIR", "").strip()
+ if raw:
+ path = Path(raw)
+ if path.is_dir():
+ return path
+ return None
+
+
+def load_reference() -> dict[str, float]:
+ """Reference distances.
+
+ Inside the unified sandbox this must come from the host benchmark dir
+ (`FRONTIER_EVAL_UNIFIED_SOURCE_BENCHMARK_DIR`; `reference.json` is never
+ copied into the sandbox). Outside the sandbox (direct run / tests) the
+ local `REFERENCE_JSON` is used as a fallback, mirroring the verification
+ evaluator so the two copies cannot drift apart.
+ """
+ candidates: list[Path] = []
+ src = _source_benchmark_dir()
+ if src is not None:
+ candidates.append(src / "data" / "reference.json")
+ env_path = os.environ.get("CVRP_EVAL_REFERENCE_JSON", "").strip()
+ if env_path:
+ candidates.append(Path(env_path))
+ candidates.append(REFERENCE_JSON)
+ for path in candidates:
+ if not path.is_file():
+ continue
+ try:
+ raw = json.loads(path.read_text(encoding="utf-8"))
+ return {k: float(v) for k, v in raw.items()}
+ except Exception:
+ continue
+ return {}
+
+
+def _initial_baseline() -> Path | None:
+ """Pristine baseline on the host, used for the fixed-region diff."""
+ src = _source_benchmark_dir()
+ if src is not None:
+ path = src / "baseline" / "solver.py"
+ if path.is_file():
+ return path
+ return None
+
+
+def _parse_env_str(name: str) -> str | None:
+ raw = os.environ.get(name, "").strip()
+ return raw or None
+
+
+def _all_instance_paths() -> list[Path]:
+ """Public instances come from the sandbox copy; held-out instances stay on
+ the host (never copied into the sandbox), so a candidate cannot read them
+ during evolution. They are still scored: the candidate receives their path
+ at scoring time (host path in process mode, repo-mount path in docker)."""
+ paths = sorted(INSTANCES_DIR.glob("*.vrp"))
+ src = _source_benchmark_dir()
+ heldout_dir = (src / "data" / "instances_heldout") if src is not None else HELDOUT_DIR
+ if heldout_dir.is_dir():
+ paths += sorted(heldout_dir.glob("*.vrp"))
+ return paths
+
+
+def _load_host_module(mod_name: str):
+ """Load a host `verification/.py` module (not present in the sandbox)."""
+ src = _source_benchmark_dir()
+ if src is None:
+ return None
+ path = src / "verification" / f"{mod_name}.py"
+ if not path.is_file():
+ return None
+ import importlib.util
+
+ spec = importlib.util.spec_from_file_location(mod_name, path)
+ mod = importlib.util.module_from_spec(spec)
+ try:
+ spec.loader.exec_module(mod)
+ except Exception:
+ return None
+ return mod
+
+
+def _reference_distance(inst: dict, ref_mod) -> float:
+ iters = max(100, 4 * inst["n"])
+ grasp_solve = ref_mod.grasp_solve
+ route_dist = ref_mod.route_dist
+ seeds = getattr(ref_mod, "REF_SEEDS", (123, 2024, 7))
+ return min(
+ sum(
+ route_dist(r, inst["distance"])
+ for r in grasp_solve(inst, starts=40, seed=s, lns_iters=iters)
+ )
+ for s in seeds
+ )
+
+
+def _generate_instances(base_seed: int, count: int, out_dir: Path) -> tuple[list[Path], dict[str, float]]:
+ """Fresh instances generated at evaluation time (anti-hardcoding).
+
+ The sandbox has no `verification/` code, so the deterministic generator and
+ reference solver are loaded from the host benchmark dir; the candidate's
+ environment is stripped of the host path (see candidate_env), so it cannot
+ reach them.
+ """
+ gen_mod = _load_host_module("generate_instances")
+ ref_mod = _load_host_module("ref_solver")
+ if gen_mod is None or ref_mod is None:
+ return [], {}
+ SPECS = gen_mod.SPECS
+ generate_instance = gen_mod.generate_instance
+ paths: list[Path] = []
+ refs: dict[str, float] = {}
+ spec_count = len(SPECS)
+ for i in range(count):
+ name, n, k, v, _seed_key = SPECS[i % spec_count]
+ fname = f"GEN-{base_seed}-{i + 1}"
+ text = generate_instance(fname, n, k, v, seed=base_seed * 1000 + i)
+ inst_path = out_dir / f"{fname}.vrp"
+ inst_path.write_text(text, encoding="ascii", newline="\n")
+ inst = parse_instance(inst_path)
+ refs[fname] = float(_reference_distance(inst, ref_mod))
+ paths.append(inst_path)
+ return paths, refs
+
+
+def _run_candidate(
+ python: str, solver_path: Path, inst_path: Path, out_path: Path, timeout: float
+) -> tuple[bool, str]:
+ try:
+ proc = subprocess.run(
+ [python, str(solver_path), str(inst_path), str(out_path)],
+ capture_output=True,
+ text=True,
+ timeout=timeout,
+ cwd=str(solver_path.parent),
+ env=candidate_env(),
+ )
+ except subprocess.TimeoutExpired:
+ return False, "timeout"
+ except OSError as exc:
+ return False, f"os error: {exc}"
+ if proc.returncode != 0:
+ return False, f"exited with code {proc.returncode}: {(proc.stderr or '')[:200]}"
+ return True, ""
+
+
+def evaluate(program_path: str, *, repo_root: Path | None = None) -> dict[str, Any]:
+ solver_path = Path(program_path).resolve()
+ if not solver_path.is_file():
+ raise FileNotFoundError(f"candidate solver not found: {solver_path}")
+
+ timeout = float(_parse_env_str("CVRP_EVAL_TIMEOUT_S") or DEFAULT_TIMEOUT_S)
+ inst_names = _parse_env_str("CVRP_EVAL_INSTANCES")
+ max_instances = _parse_env_str("CVRP_EVAL_MAX_INSTANCES")
+
+ inst_paths = _all_instance_paths()
+ if inst_names:
+ picked = [
+ p
+ for n in inst_names.replace(",", " ").split()
+ for p in inst_paths
+ if p.stem == n
+ ]
+ if picked:
+ inst_paths = picked
+ if max_instances:
+ inst_paths = inst_paths[: int(max_instances)]
+
+ reference = load_reference()
+ python = sys.executable
+ tmp_dir = Path(tempfile.mkdtemp(prefix="cvrp_eval_"))
+
+ # Anti-hardcoding: when a generation seed is provided, add fresh instances
+ # generated at evaluation time (the candidate cannot have memorized them).
+ gen_seed_raw = _parse_env_str("CVRP_EVAL_GENERATE_SEED")
+ if gen_seed_raw:
+ try:
+ gen_count = int(_parse_env_str("CVRP_EVAL_GENERATE_COUNT") or 6)
+ gen_count = max(0, gen_count)
+ except ValueError:
+ gen_count = 6
+ if gen_count > 0:
+ gen_paths, gen_refs = _generate_instances(int(gen_seed_raw), gen_count, tmp_dir)
+ inst_paths = list(inst_paths) + gen_paths
+ reference.update(gen_refs)
+
+ # Preflight: static integrity + determinism probe on the smallest instance.
+ try:
+ src_text = solver_path.read_text(encoding="utf-8", errors="replace")
+ except Exception as exc:
+ return {"metrics": {"combined_score": 0.0, "valid": 0.0}, "artifacts": {}}
+ baseline_path = _initial_baseline()
+ baseline_src = None
+ if baseline_path is not None:
+ try:
+ baseline_src = baseline_path.read_text(encoding="utf-8", errors="replace")
+ except Exception:
+ baseline_src = None
+ preflight = static_check_source(src_text, baseline_src)
+ if not preflight and inst_paths:
+ for probe in select_determinism_probes(inst_paths, parse_instance):
+ det_ok, det_note = check_determinism(python, solver_path, probe, timeout)
+ if not det_ok:
+ preflight = [f"determinism check failed on {probe.stem}: {det_note}"]
+ break
+
+ rows = []
+ for inst_path in inst_paths:
+ inst = parse_instance(inst_path)
+ out_path = tmp_dir / f"{inst['name']}.out.json"
+ ok = True
+ err = ""
+ if preflight:
+ ok = False
+ err = "preflight: " + "; ".join(preflight)
+ else:
+ ok, err = _run_candidate(python, solver_path, inst_path, out_path, timeout)
+ score = 0.0
+ cand_dist = None
+ valid = False
+ note = err if not ok else ""
+ if ok:
+ try:
+ routes = json.loads(out_path.read_text(encoding="utf-8"))
+ valid, note, cand_dist = validate(routes, inst)
+ except Exception as exc:
+ note = f"output parse error: {exc}"
+ if valid and cand_dist is not None and cand_dist > 0:
+ ref = reference.get(inst["name"])
+ if ref is not None and ref > 0:
+ scale = float(os.environ.get("CVRP_EVAL_SCORE_SCALE", "1.0"))
+ score = min(100.0, scale * 100.0 * ref / cand_dist)
+ else:
+ score = 0.0
+ valid = False
+ note = f"no reference distance for {inst['name']}"
+ rows.append(
+ {
+ "name": inst["name"],
+ "valid": bool(valid),
+ "score": score,
+ "candidate_distance": cand_dist,
+ "note": note or "",
+ }
+ )
+
+ all_valid = all(r["valid"] for r in rows)
+ combined = (
+ statistics.fmean(r["score"] for r in rows) if rows and all_valid else 0.0
+ )
+ metrics: dict[str, Any] = {
+ "combined_score": float(combined),
+ "valid": 1.0 if all_valid and rows else 0.0,
+ "instances": float(len(rows)),
+ "per_instance": {
+ r["name"]: {"score": r["score"], "valid": r["valid"], "note": r["note"]}
+ for r in rows
+ },
+ }
+ artifacts: dict[str, Any] = {
+ "candidate_path": str(solver_path),
+ "timeout_s": timeout,
+ # Reference distances are deliberately NOT included (no leak).
+ "reference_instance_count": float(len(reference)),
+ }
+ return {"metrics": metrics, "artifacts": artifacts}
diff --git a/benchmarks/VehicleRouting/CVRP/frontier_eval/initial_program.txt b/benchmarks/VehicleRouting/CVRP/frontier_eval/initial_program.txt
new file mode 100644
index 00000000..6645b02f
--- /dev/null
+++ b/benchmarks/VehicleRouting/CVRP/frontier_eval/initial_program.txt
@@ -0,0 +1 @@
+baseline/solver.py
diff --git a/benchmarks/VehicleRouting/CVRP/frontier_eval/readonly_files.txt b/benchmarks/VehicleRouting/CVRP/frontier_eval/readonly_files.txt
new file mode 100644
index 00000000..ffd0453b
--- /dev/null
+++ b/benchmarks/VehicleRouting/CVRP/frontier_eval/readonly_files.txt
@@ -0,0 +1,5 @@
+README.md
+Task.md
+verification
+data
+frontier_eval
diff --git a/benchmarks/VehicleRouting/CVRP/frontier_eval/run_eval.py b/benchmarks/VehicleRouting/CVRP/frontier_eval/run_eval.py
new file mode 100644
index 00000000..720d0e09
--- /dev/null
+++ b/benchmarks/VehicleRouting/CVRP/frontier_eval/run_eval.py
@@ -0,0 +1,126 @@
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+import traceback
+from importlib.util import module_from_spec, spec_from_file_location
+from pathlib import Path
+from typing import Any
+
+INVALID_COMBINED_SCORE = -1e18
+
+
+def _write_json(path: Path, obj: Any) -> None:
+ path.parent.mkdir(parents=True, exist_ok=True)
+ path.write_text(
+ json.dumps(obj, ensure_ascii=False, indent=2, default=str) + "\n",
+ encoding="utf-8",
+ )
+
+
+def _normalize_result(result: Any) -> tuple[dict[str, Any], dict[str, Any]]:
+ if hasattr(result, "metrics") and hasattr(result, "artifacts"):
+ return dict(getattr(result, "metrics")), dict(getattr(result, "artifacts"))
+
+ if isinstance(result, dict):
+ raw_metrics = result.get("metrics")
+ raw_artifacts = result.get("artifacts")
+ if isinstance(raw_metrics, dict):
+ return dict(raw_metrics), dict(raw_artifacts or {})
+ return dict(result), {}
+
+ raise TypeError(
+ "Evaluator must return an EvaluationResult-like object or a dict of metrics."
+ )
+
+
+def _load_local_evaluator() -> Any:
+ evaluator_path = Path(__file__).with_name("evaluator.py").resolve()
+ spec = spec_from_file_location("_frontier_eval_local_evaluator", evaluator_path)
+ if spec is None or spec.loader is None:
+ raise RuntimeError(f"Failed to load local evaluator from {evaluator_path}")
+ module = module_from_spec(spec)
+ spec.loader.exec_module(module)
+ try:
+ return getattr(module, "evaluate")
+ except AttributeError as exc:
+ raise RuntimeError(
+ f"Local evaluator does not define evaluate(): {evaluator_path}"
+ ) from exc
+
+
+def _find_repo_root() -> Path:
+ import os
+
+ env_root = os.environ.get("FRONTIER_ENGINEERING_ROOT")
+ if env_root:
+ return Path(env_root).expanduser().resolve()
+
+ here = Path(__file__).resolve()
+ for parent in [here.parent, *here.parents]:
+ if (parent / "frontier_eval").is_dir() and (parent / "benchmarks").is_dir():
+ return parent
+ return Path.cwd().resolve()
+
+
+def _build_kwargs(evaluate_fn: Any) -> dict[str, Any]:
+ kwargs: dict[str, Any] = {}
+ try:
+ parameters = inspect_signature(evaluate_fn)
+ except Exception:
+ return kwargs
+
+ if "repo_root" in parameters:
+ kwargs["repo_root"] = _find_repo_root()
+ return kwargs
+
+
+def inspect_signature(fn: Any) -> set[str]:
+ import inspect
+
+ return set(inspect.signature(fn).parameters)
+
+
+def _parse_args(argv: list[str]) -> argparse.Namespace:
+ parser = argparse.ArgumentParser(
+ description="Run a benchmark-local unified evaluator and export metrics/artifacts JSON."
+ )
+ parser.add_argument("--candidate", required=True)
+ parser.add_argument("--metrics-out", default="metrics.json")
+ parser.add_argument("--artifacts-out", default="artifacts.json")
+ return parser.parse_args(argv)
+
+
+def main(argv: list[str]) -> int:
+ args = _parse_args(argv)
+
+ candidate_path = Path(args.candidate).expanduser().resolve()
+ metrics_out = Path(args.metrics_out).expanduser().resolve()
+ artifacts_out = Path(args.artifacts_out).expanduser().resolve()
+
+ metrics: dict[str, Any] = {
+ "combined_score": INVALID_COMBINED_SCORE,
+ "valid": 0.0,
+ }
+ artifacts: dict[str, Any] = {
+ "local_evaluator_path": str(Path(__file__).with_name("evaluator.py").resolve()),
+ "candidate_path": str(candidate_path),
+ }
+
+ try:
+ evaluate_fn = _load_local_evaluator()
+ result = evaluate_fn(str(candidate_path), **_build_kwargs(evaluate_fn))
+ metrics, evaluator_artifacts = _normalize_result(result)
+ artifacts.update(evaluator_artifacts)
+ except Exception as exc:
+ artifacts["error_message"] = str(exc)
+ artifacts["traceback"] = traceback.format_exc()
+
+ _write_json(metrics_out, metrics)
+ _write_json(artifacts_out, artifacts)
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main(sys.argv[1:]))
diff --git a/benchmarks/VehicleRouting/CVRP/verification/docker/Dockerfile b/benchmarks/VehicleRouting/CVRP/verification/docker/Dockerfile
new file mode 100644
index 00000000..ec60ad3a
--- /dev/null
+++ b/benchmarks/VehicleRouting/CVRP/verification/docker/Dockerfile
@@ -0,0 +1,12 @@
+FROM python:3.11-slim
+
+WORKDIR /workspace
+
+# The evaluator, reference solver and baseline use only the Python standard
+# library. The unified runtime mounts the benchmark sandbox into the
+# container, so no benchmark files are baked into the image (this keeps
+# reference.json and the reference solver out of the container image).
+
+ENV PYTHONUNBUFFERED=1
+
+CMD ["python"]
diff --git a/benchmarks/VehicleRouting/CVRP/verification/evaluator.py b/benchmarks/VehicleRouting/CVRP/verification/evaluator.py
new file mode 100644
index 00000000..0bdc1575
--- /dev/null
+++ b/benchmarks/VehicleRouting/CVRP/verification/evaluator.py
@@ -0,0 +1,439 @@
+"""CVRP verification evaluator.
+
+Runs a candidate solver program on a fixed set of CVRP instances (public +
+held-out), validates the produced routes (full coverage, no duplicates,
+capacity respected) and scores each instance relative to the precomputed
+reference distance:
+
+ score = min(100, 100 * reference_distance / candidate_distance)
+
+An invalid/crashing/timing-out candidate scores 0 on that instance and marks
+the whole run invalid.
+
+Security / integrity:
+ * The candidate is checked before running: the EVOLVE-BLOCK markers must be
+ present, code outside the markers must match the initial baseline, and the
+ source must not reference the verification module, the reference solver,
+ reference.json, absolute filesystem paths, or hardcode per-instance routes.
+ * The candidate subprocess runs with an environment stripped of
+ FRONTIER_EVAL_UNIFIED_* variables and any reference-distance settings, so
+ it cannot learn where the scoring baseline lives.
+ * reference.json is read from the host benchmark directory (via
+ FRONTIER_EVAL_UNIFIED_SOURCE_BENCHMARK_DIR when running inside the unified
+ sandbox) and is NOT copied into the sandbox.
+
+Usage (inside the CVRP task directory):
+ python verification/evaluator.py baseline/solver.py
+ python verification/evaluator.py baseline/solver.py --instances VRP-19-2 VRP-32-5
+
+Environment overrides:
+ CVRP_EVAL_TIMEOUT_S per-instance subprocess timeout (default 60)
+ CVRP_EVAL_INSTANCES space/comma separated subset of instance names
+ CVRP_EVAL_MAX_INSTANCES cap on number of instances
+ CVRP_EVAL_SCORE_SCALE score knob (default 1.0)
+ CVRP_EVAL_REFERENCE_JSON override path to reference.json (testing only)
+ CVRP_EVAL_GENERATE_SEED when set, additionally generate fresh instances
+ from this seed at evaluation time (anti-hardcoding:
+ the candidate cannot have seen them); each generated
+ instance is scored against a reference computed on
+ the fly by the reference solver
+ CVRP_EVAL_GENERATE_COUNT number of generated instances (default 6)
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import math
+import os
+import statistics
+import subprocess
+import sys
+import tempfile
+import time
+import traceback
+from pathlib import Path
+from typing import Any
+
+TASK_ROOT = Path(__file__).resolve().parents[1] # /benchmarks/VehicleRouting/CVRP
+INSTANCES_DIR = TASK_ROOT / "data" / "instances"
+HELDOUT_DIR = TASK_ROOT / "data" / "instances_heldout"
+REFERENCE_JSON = TASK_ROOT / "data" / "reference.json"
+BASELINE_PATH = TASK_ROOT / "baseline" / "solver.py"
+
+# Candidate integrity checks live in validator.py (shared with tests).
+from validator import ( # noqa: E402
+ EVOLVE_START,
+ EVOLVE_END,
+ candidate_env,
+ check_candidate,
+ check_determinism,
+ select_determinism_probes,
+ split_evolve_blocks,
+)
+
+# Runtime instance generation (anti-hardcoding) reuses the deterministic
+# instance generator and the reference solver; see the design note in
+# investigating/docs/CVRP_heldout_记忆漏洞_问题与修复方案.md.
+from generate_instances import SPECS, generate_instance # noqa: E402
+from ref_solver import grasp_solve, route_dist # noqa: E402
+
+# Same fixed seed list the reference solver uses when producing reference.json.
+REF_SEEDS = (123, 2024, 7)
+DEFAULT_GENERATE_COUNT = 6
+
+# Backward-compatible alias used by the unit tests.
+_split_evolve_blocks = split_evolve_blocks
+
+DEFAULT_TIMEOUT_S = 60
+
+
+def parse_instance(path: Path) -> dict:
+ """Parse a TSPLIB-style CVRP .vrp file into an instance dict."""
+ text = path.read_text(encoding="utf-8", errors="ignore")
+ coords: dict[int, tuple[float, float]] = {}
+ demands: dict[int, int] = {}
+ capacity = 0
+ section = None
+ for line in text.splitlines():
+ line = line.strip()
+ if not line:
+ continue
+ upper = line.upper()
+ if upper.startswith("CAPACITY"):
+ capacity = int(line.split(":")[-1].strip())
+ continue
+ if upper == "NODE_COORD_SECTION":
+ section = "coords"
+ continue
+ if upper == "DEMAND_SECTION":
+ section = "demand"
+ continue
+ if upper == "DEPOT_SECTION":
+ section = None
+ continue
+ if upper == "EOF" or upper.startswith(("EDGE_WEIGHT", "DISPLAY_DATA")):
+ section = None
+ continue
+ if upper.startswith(("NAME", "COMMENT", "TYPE", "DIMENSION")):
+ continue
+ if section == "coords":
+ parts = line.split()
+ if len(parts) >= 3:
+ coords[int(parts[0])] = (float(parts[1]), float(parts[2]))
+ elif section == "demand":
+ parts = line.split()
+ if len(parts) >= 2:
+ demands[int(parts[0])] = int(parts[1])
+ n_customers = max(coords) - 1 # depot is id 1, customers are ids 2..n+1
+ pts = [coords[1]] + [coords[i] for i in range(2, n_customers + 2)]
+ dist = [[0] * (n_customers + 1) for _ in range(n_customers + 1)]
+ for i in range(n_customers + 1):
+ for j in range(n_customers + 1):
+ dx = pts[i][0] - pts[j][0]
+ dy = pts[i][1] - pts[j][1]
+ dist[i][j] = int(round(math.hypot(dx, dy)))
+ return {
+ "name": path.stem,
+ "n": n_customers,
+ "capacity": capacity,
+ "demand": [0] + [demands.get(i, 0) for i in range(2, n_customers + 2)],
+ "distance": dist,
+ }
+
+
+def route_distance(routes: list, dist: list[list[int]]) -> int:
+ total = 0
+ for seq in routes:
+ if not seq:
+ continue
+ total += dist[0][seq[0]]
+ for a, b in zip(seq, seq[1:]):
+ total += dist[a][b]
+ total += dist[seq[-1]][0]
+ return total
+
+
+def validate(routes: Any, inst: dict) -> tuple[bool, str, int | None]:
+ """Return (ok, error_message, total_distance)."""
+ n, cap, demand = inst["n"], inst["capacity"], inst["demand"]
+ if not isinstance(routes, list):
+ return False, "routes is not a list", None
+ seen: list[int] = []
+ for seq in routes:
+ if not isinstance(seq, list):
+ return False, "a route is not a list", None
+ load = 0
+ for c in seq:
+ if not isinstance(c, int) or isinstance(c, bool):
+ return False, f"route contains non-integer {c!r}", None
+ if c < 1 or c > n:
+ return False, f"customer id {c} out of range 1..{n}", None
+ if c in seen:
+ return False, f"customer {c} visited more than once", None
+ seen.append(c)
+ load += demand[c]
+ if load > cap:
+ return False, f"route {seq} exceeds capacity {cap} (load {load})", None
+ if set(seen) != set(range(1, n + 1)):
+ missing = set(range(1, n + 1)) - set(seen)
+ return False, f"customers not served: {sorted(missing)}", None
+ return True, "", route_distance(routes, inst["distance"])
+
+
+def run_candidate(
+ python: str, solver_path: Path, inst_path: Path, out_path: Path, timeout: float
+) -> tuple[bool, str]:
+ try:
+ proc = subprocess.run(
+ [python, str(solver_path), str(inst_path), str(out_path)],
+ capture_output=True,
+ text=True,
+ timeout=timeout,
+ cwd=str(solver_path.parent),
+ env=candidate_env(),
+ )
+ except subprocess.TimeoutExpired:
+ return False, "timeout"
+ except OSError as exc:
+ return False, f"os error: {exc}"
+ if proc.returncode != 0:
+ return False, f"exited with code {proc.returncode}: {(proc.stderr or '')[:200]}"
+ return True, ""
+
+
+def _source_benchmark_dir() -> Path | None:
+ """Host benchmark dir exposed by the unified sandbox (if running there)."""
+ raw = os.environ.get("FRONTIER_EVAL_UNIFIED_SOURCE_BENCHMARK_DIR", "").strip()
+ if raw:
+ path = Path(raw)
+ if path.is_dir():
+ return path
+ return None
+
+
+def load_reference() -> dict[str, float]:
+ """Reference distances, preferring the host copy over the sandbox copy."""
+ candidates: list[Path] = []
+ src = _source_benchmark_dir()
+ if src is not None:
+ candidates.append(src / "data" / "reference.json")
+ env_path = os.environ.get("CVRP_EVAL_REFERENCE_JSON", "").strip()
+ if env_path:
+ candidates.append(Path(env_path))
+ candidates.append(REFERENCE_JSON)
+ for path in candidates:
+ if not path.is_file():
+ continue
+ try:
+ raw = json.loads(path.read_text(encoding="utf-8"))
+ return {k: float(v) for k, v in raw.items()}
+ except Exception:
+ continue
+ return {}
+
+
+def _parse_env_str(name: str) -> str | None:
+ raw = __import__("os").environ.get(name, "").strip()
+ return raw or None
+
+
+def _all_instance_paths() -> list[Path]:
+ """Public instances come from the sandbox copy; held-out instances stay on
+ the host (never copied into the sandbox), so a candidate cannot read them
+ during evolution. They are still scored: the candidate receives their path
+ at scoring time (host path in process mode, repo-mount path in docker)."""
+ paths = sorted(INSTANCES_DIR.glob("*.vrp"))
+ src = _source_benchmark_dir()
+ heldout_dir = (src / "data" / "instances_heldout") if src is not None else HELDOUT_DIR
+ if heldout_dir.is_dir():
+ paths += sorted(heldout_dir.glob("*.vrp"))
+ return paths
+
+
+def _reference_distance(inst: dict) -> float:
+ """Deterministic reference distance for an instance (GRASP + LNS, fixed seeds)."""
+ iters = max(100, 4 * inst["n"])
+ return min(
+ sum(
+ route_dist(r, inst["distance"])
+ for r in grasp_solve(inst, starts=40, seed=seed, lns_iters=iters)
+ )
+ for seed in REF_SEEDS
+ )
+
+
+def _generate_instances(base_seed: int, count: int, out_dir: Path) -> tuple[list[Path], dict[str, float]]:
+ """Generate fresh instances from `base_seed`; return (instance files, ref distances).
+
+ Instance sizes follow the public set's SPECS (cycled), and per-instance RNG
+ streams are derived as `base_seed * 1000 + i`, matching multiseed_stat.py.
+ Deterministic: same seed + same machine yields the same instances and refs.
+ """
+ paths: list[Path] = []
+ refs: dict[str, float] = {}
+ spec_count = len(SPECS)
+ for i in range(count):
+ name, n, k, v, _seed_key = SPECS[i % spec_count]
+ fname = f"GEN-{base_seed}-{i + 1}"
+ text = generate_instance(fname, n, k, v, seed=base_seed * 1000 + i)
+ inst_path = out_dir / f"{fname}.vrp"
+ inst_path.write_text(text, encoding="ascii", newline="\n")
+ inst = parse_instance(inst_path)
+ refs[fname] = float(_reference_distance(inst))
+ paths.append(inst_path)
+ return paths, refs
+
+
+def evaluate(program_path: str, *, repo_root: Path | None = None) -> dict[str, Any]:
+ solver_path = Path(program_path).resolve()
+ if not solver_path.is_file():
+ raise FileNotFoundError(f"candidate solver not found: {solver_path}")
+
+ timeout = float(_parse_env_str("CVRP_EVAL_TIMEOUT_S") or DEFAULT_TIMEOUT_S)
+ inst_names = _parse_env_str("CVRP_EVAL_INSTANCES")
+ max_instances = _parse_env_str("CVRP_EVAL_MAX_INSTANCES")
+
+ inst_paths = _all_instance_paths()
+ if inst_names:
+ picked = [p for n in inst_names.replace(",", " ").split() for p in inst_paths if p.stem == n]
+ if picked:
+ inst_paths = picked
+ if max_instances:
+ inst_paths = inst_paths[: int(max_instances)]
+
+ reference = load_reference()
+ python = sys.executable
+ tmp_dir = Path(tempfile.mkdtemp(prefix="cvrp_eval_"))
+
+ # Anti-hardcoding: when a generation seed is provided, add fresh instances
+ # generated at evaluation time (the candidate cannot have memorized them).
+ gen_seed_raw = _parse_env_str("CVRP_EVAL_GENERATE_SEED")
+ if gen_seed_raw:
+ try:
+ gen_count = int(_parse_env_str("CVRP_EVAL_GENERATE_COUNT") or DEFAULT_GENERATE_COUNT)
+ gen_count = max(0, gen_count)
+ except ValueError:
+ gen_count = DEFAULT_GENERATE_COUNT
+ if gen_count > 0:
+ gen_paths, gen_refs = _generate_instances(int(gen_seed_raw), gen_count, tmp_dir)
+ inst_paths = list(inst_paths) + gen_paths
+ reference.update(gen_refs)
+
+ # Static integrity checks + determinism (probe small / medium / large
+ # instances; each probe runs the candidate twice).
+ src_dir = _source_benchmark_dir()
+ baseline_path = (
+ (src_dir / "baseline" / "solver.py") if src_dir is not None else BASELINE_PATH
+ )
+ preflight = check_candidate(solver_path, baseline_path=baseline_path)
+ if not preflight and inst_paths:
+ for probe in select_determinism_probes(inst_paths, parse_instance):
+ det_ok, det_note = check_determinism(
+ python, solver_path, probe, timeout
+ )
+ if not det_ok:
+ preflight = [
+ f"determinism check failed on {probe.stem}: {det_note}"
+ ]
+ break
+
+ rows = []
+ for inst_path in inst_paths:
+ inst = parse_instance(inst_path)
+ out_path = tmp_dir / f"{inst['name']}.out.json"
+ ok = True
+ err = ""
+ if preflight:
+ ok = False
+ err = "preflight: " + "; ".join(preflight)
+ else:
+ ok, err = run_candidate(python, solver_path, inst_path, out_path, timeout)
+ score = 0.0
+ cand_dist = None
+ valid = False
+ note = err if not ok else ""
+ if ok:
+ try:
+ routes = json.loads(out_path.read_text(encoding="utf-8"))
+ valid, note, cand_dist = validate(routes, inst)
+ except Exception as exc:
+ note = f"output parse error: {exc}"
+ if valid and cand_dist is not None and cand_dist > 0:
+ ref = reference.get(inst["name"])
+ if ref is not None and ref > 0:
+ # Score scale: experimental knob to tighten the 100-point bar.
+ # Default 1.0 => score = min(100, 100*ref/cand). A scale s means
+ # a candidate must be s times shorter than the reference to hit
+ # 100 (e.g. s=2/3 => needs cand <= ref*2/3).
+ scale = float(os.environ.get("CVRP_EVAL_SCORE_SCALE", "1.0"))
+ score = min(100.0, scale * 100.0 * ref / cand_dist)
+ else:
+ score = 0.0
+ valid = False
+ note = f"no reference distance for {inst['name']}"
+ rows.append(
+ {
+ "name": inst["name"],
+ "valid": bool(valid),
+ "score": score,
+ "candidate_distance": cand_dist,
+ "note": note or "",
+ }
+ )
+
+ all_valid = all(r["valid"] for r in rows)
+ combined = (
+ statistics.fmean(r["score"] for r in rows) if rows and all_valid else 0.0
+ )
+ metrics: dict[str, Any] = {
+ "combined_score": float(combined),
+ "valid": 1.0 if all_valid and rows else 0.0,
+ "instances": float(len(rows)),
+ "per_instance": {
+ r["name"]: {"score": r["score"], "valid": r["valid"], "note": r["note"]}
+ for r in rows
+ },
+ }
+ artifacts = {
+ "candidate_path": str(solver_path),
+ "timeout_s": timeout,
+ # NOTE: reference distances are deliberately NOT included here to avoid
+ # leaking the scoring baseline to the agent.
+ "reference_instance_count": float(len(reference)),
+ }
+ return {"metrics": metrics, "artifacts": artifacts}
+
+
+def _print_report(metrics: dict[str, Any]) -> None:
+ print(f"combined_score: {metrics['combined_score']:.2f}")
+ print(f"valid: {metrics['valid']}")
+ for name, info in metrics.get("per_instance", {}).items():
+ flag = "OK " if info["valid"] else "BAD"
+ note = f" ({info['note']})" if info.get("note") else ""
+ print(f" {flag} {name}: score={info['score']:.2f}{note}")
+
+
+def main(argv: list[str] | None = None) -> int:
+ parser = argparse.ArgumentParser(description="CVRP candidate evaluator")
+ parser.add_argument("candidate", help="path to candidate solver.py")
+ parser.add_argument("--instances", nargs="*", default=None)
+ args = parser.parse_args(argv)
+
+ os_env = __import__("os").environ
+ if args.instances:
+ os_env["CVRP_EVAL_INSTANCES"] = " ".join(args.instances)
+
+ t0 = time.perf_counter()
+ try:
+ result = evaluate(args.candidate)
+ metrics, artifacts = result["metrics"], result["artifacts"]
+ _print_report(metrics)
+ print(f"wall time: {time.perf_counter() - t0:.1f}s")
+ return 0
+ except Exception:
+ print(traceback.format_exc(), file=sys.stderr)
+ return 1
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/benchmarks/VehicleRouting/CVRP/verification/generate_instances.py b/benchmarks/VehicleRouting/CVRP/verification/generate_instances.py
new file mode 100644
index 00000000..a192de0f
--- /dev/null
+++ b/benchmarks/VehicleRouting/CVRP/verification/generate_instances.py
@@ -0,0 +1,145 @@
+"""Generate reproducible CVRP instances (clustered Euclidean, TSPLIB format).
+
+Usage:
+ python verification/generate_instances.py
+
+Writes TSPLIB-style .vrp files under data/instances/.
+Instances are deterministic (fixed seed) so results are reproducible everywhere.
+"""
+from __future__ import annotations
+
+import math
+import random
+from pathlib import Path
+
+SEED = 42
+OUT_DIR = Path(__file__).resolve().parents[1] / "data" / "instances"
+HELDOUT_DIR = Path(__file__).resolve().parents[1] / "data" / "instances_heldout"
+
+# (name, num_customers, num_clusters, num_vehicles_approx, seed_key)
+# `seed_key` fixes the RNG stream per instance. The keys are the original
+# G-* identifiers: the dataset was first released under those names, and
+# keeping the keys means the coordinates/demands stay byte-identical to the
+# validated release. Do not change them.
+SPECS = [
+ ("VRP-19-2", 19, 2, 2, "G-n19-k2"),
+ ("VRP-21-3", 21, 3, 3, "G-n21-k3"),
+ ("VRP-22-4", 22, 3, 4, "G-n22-k4"),
+ ("VRP-32-5", 32, 4, 5, "G-n32-k5"),
+ ("VRP-37-6", 37, 4, 6, "G-n37-k6"),
+ ("VRP-45-6", 45, 5, 6, "G-n45-k6"),
+ ("VRP-45-7", 45, 5, 7, "G-n45-k7"),
+ ("VRP-48-7", 48, 5, 7, "G-n48-k7"),
+ ("VRP-54-8", 54, 6, 8, "G-n54-k8"),
+ ("VRP-55-8", 55, 6, 8, "G-n55-k8"),
+ ("VRP-60-9", 60, 6, 9, "G-n60-k9"),
+ ("VRP-60-10", 60, 6, 10, "G-n60-k10"),
+]
+
+# Held-out instances: used only at evaluation time, never shown to the agent
+# (they are absent from agent_files and Task.md), so a candidate cannot hardcode
+# routes by instance name. Names use a distinct VHO- prefix.
+HELDOUT_SPECS = [
+ ("VHO-22-3", 22, 3, 3),
+ ("VHO-24-3", 24, 3, 3),
+ ("VHO-27-4", 27, 3, 4),
+ ("VHO-29-4", 29, 4, 4),
+ ("VHO-33-5", 33, 4, 5),
+ ("VHO-36-5", 36, 4, 5),
+ ("VHO-39-6", 39, 5, 6),
+ ("VHO-43-6", 43, 5, 6),
+ ("VHO-47-7", 47, 5, 7),
+ ("VHO-51-7", 51, 6, 7),
+ ("VHO-55-8", 55, 6, 8),
+ ("VHO-58-9", 58, 6, 9),
+]
+
+
+def _round_dist(x1: float, y1: float, x2: float, y2: float) -> int:
+ return int(round(math.hypot(x1 - x2, y1 - y2)))
+
+
+def generate_instance(
+ name: str,
+ num_customers: int,
+ num_clusters: int,
+ vehicles: int,
+ seed_key: str | None = None,
+ seed: int | None = None,
+) -> str:
+ """Generate one deterministic CVRP instance as TSPLIB text.
+
+ Exactly one of `seed_key` (legacy: fixed base seed + key, e.g. "G-n19-k2")
+ or `seed` (arbitrary integer) must be provided; `seed` lets callers derive
+ fresh instances at evaluation time without touching the released dataset.
+ """
+ if seed is not None:
+ rng = random.Random(seed)
+ else:
+ assert seed_key is not None, "provide either seed_key or seed"
+ rng = random.Random(f"{SEED}:{seed_key}")
+ # Cluster centers spread over a 100x100 region.
+ centers = [
+ (rng.uniform(15, 85), rng.uniform(15, 85)) for _ in range(num_clusters)
+ ]
+ # Depot at a central-ish position.
+ depot = (50.0, 50.0)
+
+ coords = [depot]
+ for i in range(num_customers):
+ cx, cy = centers[i % len(centers)]
+ x = min(100, max(1, round(cx + rng.gauss(0, 7.0))))
+ y = min(100, max(1, round(cy + rng.gauss(0, 7.0))))
+ coords.append((x, y))
+
+ demand = [0] + [rng.randint(5, 35) for _ in range(num_customers)]
+ total_demand = sum(demand)
+ # Capacity tuned so the instance is feasible with `vehicles` vehicles.
+ capacity = max(30, math.ceil(total_demand / vehicles * 1.25 / 5) * 5)
+
+ dim = num_customers + 1
+ lines = [
+ f"NAME: {name}",
+ "COMMENT: generated clustered CVRP instance (deterministic)",
+ "TYPE: CVRP",
+ f"DIMENSION: {dim}",
+ f"CAPACITY: {capacity}",
+ "NODE_COORD_SECTION",
+ ]
+ for idx, (x, y) in enumerate(coords, start=1):
+ lines.append(f"{idx} {x} {y}")
+ lines.append("DEMAND_SECTION")
+ for idx in range(1, dim + 1):
+ lines.append(f"{idx} {demand[idx - 1]}")
+ lines.append("DEPOT_SECTION")
+ lines.append("1")
+ lines.append("EOF")
+ return "\n".join(lines) + "\n"
+
+
+def main() -> None:
+ OUT_DIR.mkdir(parents=True, exist_ok=True)
+ for name, n, k, v, seed_key in SPECS:
+ text = generate_instance(name, n, k, v, seed_key)
+ (OUT_DIR / f"{name}.vrp").write_text(
+ text, encoding="ascii", newline="\n" # force LF: byte-identical on any OS
+ )
+ print(
+ f"wrote {name}.vrp (customers={n}, "
+ f"capacity={text.split('CAPACITY: ')[1].split()[0]})"
+ )
+
+ HELDOUT_DIR.mkdir(parents=True, exist_ok=True)
+ for name, n, k, v in HELDOUT_SPECS:
+ text = generate_instance(name, n, k, v, seed_key=f"HO-{name}")
+ (HELDOUT_DIR / f"{name}.vrp").write_text(
+ text, encoding="ascii", newline="\n"
+ )
+ print(
+ f"wrote heldout {name}.vrp (customers={n}, "
+ f"capacity={text.split('CAPACITY: ')[1].split()[0]})"
+ )
+
+
+if __name__ == "__main__":
+ main()
diff --git a/benchmarks/VehicleRouting/CVRP/verification/multiseed_stat.py b/benchmarks/VehicleRouting/CVRP/verification/multiseed_stat.py
new file mode 100644
index 00000000..476f11a9
--- /dev/null
+++ b/benchmarks/VehicleRouting/CVRP/verification/multiseed_stat.py
@@ -0,0 +1,111 @@
+"""Multi-seed statistics for the CVRP baseline.
+
+The released evaluation set (24 instances: 12 public + 12 held-out) is fixed,
+so the baseline solver is deterministic on it. To satisfy "multi-seed /
+multi-run statistics" (reviewer request), this script derives *fresh*
+evaluation instance sets from multiple seeds (via
+`generate_instances.generate_instance(..., seed=...)`), computes each set's
+reference distances on the fly (deterministic GRASP + LNS), and reports the
+baseline score distribution across seeds.
+
+Usage (inside the CVRP task directory):
+ python verification/multiseed_stat.py [--seeds 111 222 333]
+
+Deterministic: same command + same machine yields the same numbers.
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import statistics
+import subprocess
+import sys
+import tempfile
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+from evaluator import validate # noqa: E402
+from generate_instances import SPECS, generate_instance # noqa: E402
+from ref_solver import grasp_solve, parse_instance, route_dist # noqa: E402
+
+CVRP_ROOT = Path(__file__).resolve().parents[1]
+BASELINE = CVRP_ROOT / "baseline" / "solver.py"
+REF_SEEDS = (123, 2024, 7) # same fixed seed list as ref_solver.main()
+
+
+def reference_distance(inst: dict) -> int:
+ iters = max(100, 4 * inst["n"])
+ return min(
+ sum(
+ route_dist(r, inst["distance"])
+ for r in grasp_solve(inst, starts=40, seed=seed, lns_iters=iters)
+ )
+ for seed in REF_SEEDS
+ )
+
+
+def run_baseline(
+ python: str, inst_path: Path, out_path: Path, timeout: float
+) -> list:
+ proc = subprocess.run(
+ [python, str(BASELINE), str(inst_path), str(out_path)],
+ capture_output=True,
+ text=True,
+ timeout=timeout,
+ cwd=str(BASELINE.parent),
+ )
+ if proc.returncode != 0:
+ raise RuntimeError(f"baseline failed on {inst_path.name}: {proc.stderr[:200]}")
+ return json.loads(out_path.read_text(encoding="utf-8"))
+
+
+def score_seed(seed: int, python: str, tmp: Path, timeout: float) -> tuple[float, list]:
+ """Derive one instance set from `seed`; return (combined, per-instance)."""
+ inst_dir = tmp / f"s{seed}"
+ inst_dir.mkdir(parents=True, exist_ok=True)
+ scores = []
+ for i, (name, n, k, v, _seed_key) in enumerate(SPECS):
+ fname = f"S{seed}-{i + 1}"
+ text = generate_instance(fname, n, k, v, seed=seed * 1000 + i)
+ inst_path = inst_dir / f"{fname}.vrp"
+ inst_path.write_text(text, encoding="ascii", newline="\n")
+
+ inst = parse_instance(inst_path)
+ ref = reference_distance(inst)
+
+ out_path = inst_dir / f"{fname}.out.json"
+ routes = run_baseline(python, inst_path, out_path, timeout)
+ ok, note, cand = validate(routes, inst)
+ if not ok:
+ raise RuntimeError(f"baseline invalid on {fname}: {note}")
+ scores.append(min(100.0, 100.0 * ref / cand))
+ return statistics.fmean(scores), scores
+
+
+def main(argv: list[str] | None = None) -> int:
+ parser = argparse.ArgumentParser(description="CVRP baseline multi-seed stats")
+ parser.add_argument("--seeds", nargs="*", type=int, default=[111, 222, 333])
+ parser.add_argument("--timeout", type=float, default=60.0)
+ args = parser.parse_args(argv)
+
+ python = sys.executable
+ with tempfile.TemporaryDirectory(prefix="cvrp_multiseed_") as td:
+ tmp = Path(td)
+ per_seed: list[tuple[int, float, list]] = []
+ for seed in args.seeds:
+ combined, scores = score_seed(seed, python, tmp, args.timeout)
+ per_seed.append((seed, combined, scores))
+ print(f"seed {seed}: combined={combined:.2f}")
+
+ combined_values = [c for _, c, _ in per_seed]
+ print(
+ f"multi-seed: mean={statistics.fmean(combined_values):.2f} "
+ f"std={statistics.pstdev(combined_values):.2f} "
+ f"min={min(combined_values):.2f} max={max(combined_values):.2f} "
+ f"seeds={args.seeds}"
+ )
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/benchmarks/VehicleRouting/CVRP/verification/ref_solver.py b/benchmarks/VehicleRouting/CVRP/verification/ref_solver.py
new file mode 100644
index 00000000..9f8fb52d
--- /dev/null
+++ b/benchmarks/VehicleRouting/CVRP/verification/ref_solver.py
@@ -0,0 +1,487 @@
+"""Reference solver for CVRP: GRASP multi-start + intra 2-opt +
+relocate/swap/2-opt* + LNS refinement with greedy repair and tabu
+diversification.
+
+Pure standard library. Used to produce data/reference.json (the score
+baseline per instance).
+
+Deterministic: every source of randomness is seeded and the LNS search
+budget is measured in *iterations*, not wall-clock time, so regenerating
+reference.json on any machine yields byte-identical output. main() runs a
+fixed list of seeds and keeps the best distance per instance.
+
+Usage:
+ python verification/ref_solver.py [--starts N] [--iterations N]
+
+ --starts N number of GRASP multi-start constructions (default 40)
+ --iterations N fixed LNS iterations per instance (default: max(100, 4*n))
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import math
+import random
+from collections import deque
+from pathlib import Path
+
+INST_DIR = Path(__file__).resolve().parents[1] / "data" / "instances"
+HELDOUT_DIR = Path(__file__).resolve().parents[1] / "data" / "instances_heldout"
+OUT_JSON = Path(__file__).resolve().parents[1] / "data" / "reference.json"
+
+
+def parse_instance(path: Path) -> dict:
+ """Parse a TSPLIB-style CVRP .vrp file."""
+ text = path.read_text(encoding="utf-8", errors="ignore")
+ coords: dict[int, tuple[float, float]] = {}
+ demands: dict[int, int] = {}
+ capacity = 0
+ section = None
+ for line in text.splitlines():
+ line = line.strip()
+ if not line:
+ continue
+ upper = line.upper()
+ if upper.startswith("CAPACITY"):
+ capacity = int(line.split(":")[-1].strip())
+ continue
+ if upper == "NODE_COORD_SECTION":
+ section = "coords"
+ continue
+ if upper == "DEMAND_SECTION":
+ section = "demand"
+ continue
+ if upper == "DEPOT_SECTION":
+ section = "depot"
+ continue
+ if upper == "EOF" or upper.startswith("EDGE_WEIGHT") or upper.startswith("DISPLAY_DATA"):
+ section = None
+ continue
+ if upper.startswith(("NAME", "COMMENT", "TYPE", "DIMENSION")):
+ continue
+ if section == "coords":
+ parts = line.split()
+ if len(parts) >= 3:
+ coords[int(parts[0])] = (float(parts[1]), float(parts[2]))
+ elif section == "demand":
+ parts = line.split()
+ if len(parts) >= 2:
+ demands[int(parts[0])] = int(parts[1])
+ n_customers = max(coords) - 1 # depot is id 1, customers are ids 2..n+1
+ pts = [coords[1]] + [coords[i] for i in range(2, n_customers + 2)]
+ dist = [[0] * (n_customers + 1) for _ in range(n_customers + 1)]
+ for i in range(n_customers + 1):
+ for j in range(n_customers + 1):
+ dx = pts[i][0] - pts[j][0]
+ dy = pts[i][1] - pts[j][1]
+ dist[i][j] = int(round(math.hypot(dx, dy)))
+ demand = [0] + [demands.get(i, 0) for i in range(2, n_customers + 2)]
+ return {
+ "n": n_customers,
+ "capacity": capacity,
+ "demand": demand,
+ "distance": dist,
+ "name": path.stem,
+ }
+
+
+def route_dist(seq: list[int], dist: list[list[int]]) -> int:
+ if not seq:
+ return 0
+ total = dist[0][seq[0]]
+ for a, b in zip(seq, seq[1:]):
+ total += dist[a][b]
+ total += dist[seq[-1]][0]
+ return total
+
+
+def two_opt(seq: list[int], dist: list[list[int]]) -> list[int]:
+ best = seq[:]
+ improved = True
+ while improved:
+ improved = False
+ for a in range(len(best) - 1):
+ for b in range(a + 1, len(best)):
+ cand = best[:a] + best[a : b + 1][::-1] + best[b + 1 :]
+ if route_dist(cand, dist) < route_dist(best, dist):
+ best = cand
+ improved = True
+ return best
+
+
+def savings_solve(inst: dict, rng: random.Random | None = None) -> list[list[int]]:
+ n, cap = inst["n"], inst["capacity"]
+ demand, dist = inst["demand"], inst["distance"]
+ routes = {c: [c] for c in range(1, n + 1)}
+ load = {c: demand[c] for c in range(1, n + 1)}
+ first = {c: c for c in range(1, n + 1)}
+ last = {c: c for c in range(1, n + 1)}
+ owner = {c: c for c in range(1, n + 1)}
+
+ savings = []
+ for i in range(1, n + 1):
+ for j in range(i + 1, n + 1):
+ savings.append((dist[0][i] + dist[0][j] - dist[i][j], i, j))
+ if rng is None:
+ savings.sort(reverse=True)
+ else:
+ # Randomized savings (GRASP-style perturbation) for multi-start search.
+ savings.sort(
+ key=lambda t: t[0] + rng.uniform(-abs(t[0]) * 0.5, abs(t[0]) * 0.5),
+ reverse=True,
+ )
+
+ for _s, i, j in savings:
+ ri, rj = owner[i], owner[j]
+ if ri == rj:
+ continue
+ if load[ri] + load[rj] > cap:
+ continue
+ # Merge rj into ri when ri's tail == i and rj's head == j (or symmetric).
+ if last[ri] == i and first[rj] == j:
+ for c in routes[rj]:
+ owner[c] = ri
+ routes[ri] = routes[ri] + routes[rj]
+ load[ri] += load[rj]
+ last[ri] = last[rj]
+ del routes[rj], load[rj], first[rj], last[rj]
+ elif last[rj] == i and first[ri] == j:
+ for c in routes[ri]:
+ owner[c] = rj
+ routes[rj] = routes[rj] + routes[ri]
+ load[rj] += load[ri]
+ last[rj] = last[ri]
+ del routes[ri], load[ri], first[ri], last[ri]
+ elif last[ri] == j and first[rj] == i:
+ for c in routes[rj]:
+ owner[c] = ri
+ routes[ri] = routes[ri] + routes[rj]
+ load[ri] += load[rj]
+ last[ri] = last[rj]
+ del routes[rj], load[rj], first[rj], last[rj]
+ elif last[rj] == j and first[ri] == i:
+ for c in routes[ri]:
+ owner[c] = rj
+ routes[rj] = routes[rj] + routes[ri]
+ load[rj] += load[ri]
+ last[rj] = last[ri]
+ del routes[ri], load[ri], first[ri], last[ri]
+
+ final = []
+ for seq in routes.values():
+ final.append(two_opt(seq, dist))
+ return final
+
+
+def relocate_improve(routes: list[list[int]], inst: dict) -> list[list[int]]:
+ """Best-improve cross-route relocate local search.
+
+ Repeatedly moves the single customer that most reduces total distance
+ (removal from its route + best insertion into another route), respecting
+ capacity, until no improving move remains.
+ """
+ n, cap, demand, dist = inst["n"], inst["capacity"], inst["demand"], inst["distance"]
+ routes = [list(r) for r in routes if r]
+ improved = True
+ while improved:
+ improved = False
+ best_gain = 0.0
+ best = None # (ri, pos, rj, c)
+ for ri in range(len(routes)):
+ route = routes[ri]
+ if not route:
+ continue
+ d_ri = route_dist(route, dist)
+ for pos in range(len(route)):
+ c = route[pos]
+ ra = route[:pos] + route[pos + 1 :]
+ d_ra = route_dist(ra, dist) if ra else 0
+ gain_remove = d_ri - d_ra
+ for rj in range(len(routes)):
+ if rj == ri:
+ continue
+ rj_route = routes[rj]
+ if not rj_route:
+ if demand[c] > cap:
+ continue
+ gain_insert = -(dist[0][c] + dist[c][0])
+ else:
+ if sum(demand[x] for x in rj_route) + demand[c] > cap:
+ continue
+ d_rj = route_dist(rj_route, dist)
+ gain_insert = max(
+ d_rj
+ - route_dist(
+ rj_route[:ipos] + [c] + rj_route[ipos:], dist
+ )
+ for ipos in range(len(rj_route) + 1)
+ )
+ total = gain_remove + gain_insert
+ if total > best_gain + 1e-9:
+ best_gain = total
+ best = (ri, pos, rj, c)
+ if best is not None:
+ ri, pos, rj, c = best
+ routes[ri].pop(pos)
+ if not routes[ri]:
+ routes.pop(ri)
+ if rj > ri:
+ rj -= 1
+ rj_route = routes[rj]
+ best_ipos = min(
+ range(len(rj_route) + 1),
+ key=lambda ipos: route_dist(rj_route[:ipos] + [c] + rj_route[ipos:], dist),
+ )
+ routes[rj].insert(best_ipos, c)
+ improved = True
+ return routes
+
+
+def swap_improve(routes: list[list[int]], inst: dict) -> list[list[int]]:
+ """Best-improve cross-route swap local search (exchange one customer each)."""
+ cap, demand, dist = inst["capacity"], inst["demand"], inst["distance"]
+ routes = [list(r) for r in routes if r]
+ improved = True
+ while improved:
+ improved = False
+ best_gain = 0.0
+ best = None # (ri, pi, rj, pj)
+ for ri in range(len(routes)):
+ for rj in range(ri + 1, len(routes)):
+ a, b = routes[ri], routes[rj]
+ load_a = sum(demand[x] for x in a)
+ load_b = sum(demand[x] for x in b)
+ d_a = route_dist(a, dist)
+ d_b = route_dist(b, dist)
+ for pi in range(len(a)):
+ ci = a[pi]
+ for pj in range(len(b)):
+ cj = b[pj]
+ if load_a - demand[ci] + demand[cj] > cap:
+ continue
+ if load_b - demand[cj] + demand[ci] > cap:
+ continue
+ new_a = a[:pi] + [cj] + a[pi + 1 :]
+ new_b = b[:pj] + [ci] + b[pj + 1 :]
+ gain = (d_a + d_b) - (
+ route_dist(new_a, dist) + route_dist(new_b, dist)
+ )
+ if gain > best_gain + 1e-9:
+ best_gain = gain
+ best = (ri, pi, rj, pj)
+ if best is not None:
+ ri, pi, rj, pj = best
+ routes[ri][pi], routes[rj][pj] = routes[rj][pj], routes[ri][pi]
+ improved = True
+ return routes
+
+
+def two_opt_star_improve(routes: list[list[int]], inst: dict) -> list[list[int]]:
+ """Best-improve cross-route 2-opt* local search (swap route tails)."""
+ cap, demand, dist = inst["capacity"], inst["demand"], inst["distance"]
+ routes = [list(r) for r in routes if r]
+ improved = True
+ while improved:
+ improved = False
+ best_gain = 0.0
+ best = None # (ri, rj, new_a, new_b)
+ for ri in range(len(routes)):
+ for rj in range(ri + 1, len(routes)):
+ a, b = routes[ri], routes[rj]
+ d_a = route_dist(a, dist)
+ d_b = route_dist(b, dist)
+ for pi in range(len(a)):
+ for pj in range(len(b)):
+ new_a = a[: pi + 1] + b[pj + 1 :]
+ new_b = b[: pj + 1] + a[pi + 1 :]
+ if sum(demand[x] for x in new_a) > cap:
+ continue
+ if sum(demand[x] for x in new_b) > cap:
+ continue
+ gain = (d_a + d_b) - (
+ route_dist(new_a, dist) + route_dist(new_b, dist)
+ )
+ if gain > best_gain + 1e-9:
+ best_gain = gain
+ best = (ri, rj, new_a, new_b)
+ if best is not None:
+ ri, rj, new_a, new_b = best
+ routes[ri] = new_a
+ routes[rj] = new_b
+ improved = True
+ return routes
+
+
+def local_search(routes: list[list[int]], inst: dict) -> list[list[int]]:
+ """Intensify a solution: intra 2-opt + Or-opt + relocate + swap + 2-opt*
+ to a fixed point."""
+ dist = inst["distance"]
+ routes = [list(r) for r in routes if r]
+ improved = True
+ while improved:
+ before = sum(route_dist(r, dist) for r in routes)
+ routes = [two_opt(r, dist) for r in routes]
+ routes = relocate_improve(routes, inst)
+ routes = swap_improve(routes, inst)
+ routes = two_opt_star_improve(routes, inst)
+ after = sum(route_dist(r, dist) for r in routes)
+ improved = after < before - 1e-9
+ return routes
+
+
+def greedy_repair(
+ routes: list[list[int]], unrouted: list[int], inst: dict, rng: random.Random
+) -> list[list[int]]:
+ """Reinsert every customer in `unrouted` at its cheapest feasible position.
+
+ Insertion order is a random permutation driven by `rng`, and a new route
+ (depot-customer-depot) is used whenever it is cheaper than any feasible
+ insertion or none exists. Deterministic for a fixed `rng`.
+ """
+ cap, demand, dist = inst["capacity"], inst["demand"], inst["distance"]
+ routes = [list(r) for r in routes if r]
+ for c in rng.sample(unrouted, len(unrouted)):
+ best_inc = None # cheapest insertion cost increase
+ best_pos = None # (route index, position)
+ for ri, r in enumerate(routes):
+ if sum(demand[x] for x in r) + demand[c] > cap:
+ continue
+ for pos in range(len(r) + 1):
+ pred = r[pos - 1] if pos > 0 else 0
+ succ = r[pos] if pos < len(r) else 0
+ inc = dist[pred][c] + dist[c][succ] - dist[pred][succ]
+ if best_inc is None or inc < best_inc - 1e-9:
+ best_inc = inc
+ best_pos = (ri, pos)
+ new_route_inc = 2 * dist[0][c]
+ if best_inc is None or new_route_inc < best_inc - 1e-9:
+ routes.append([c])
+ else:
+ ri, pos = best_pos
+ routes[ri].insert(pos, c)
+ return routes
+
+
+def lns_improve(
+ routes: list[list[int]],
+ inst: dict,
+ rng: random.Random,
+ max_iters: int,
+ tabu_size: int = 12,
+) -> list[list[int]]:
+ """Large neighbourhood search with tabu diversification.
+
+ Repeat (deterministically, seeded by `rng`) `max_iters` times: destroy
+ 25%-50% of customers, reinsert them with `greedy_repair`, then intensify
+ with `local_search`. Destroy sets recently used are skipped (tabu list),
+ so the search keeps exploring. Returns the best solution found.
+ """
+ n, cap, demand, dist = inst["n"], inst["capacity"], inst["demand"], inst["distance"]
+ best = [list(r) for r in routes if r]
+ best_cost = sum(route_dist(r, dist) for r in best)
+ tabu: deque[tuple[int, ...]] = deque()
+ tabu_set: set[tuple[int, ...]] = set()
+
+ for _ in range(max_iters):
+ remove = None
+ for _attempt in range(8):
+ q = rng.randint(max(1, n // 4), max(1, n // 2))
+ cand = tuple(sorted(rng.sample(range(1, n + 1), q)))
+ if cand not in tabu_set:
+ remove = cand
+ break
+ if remove is None:
+ remove = tuple(sorted(rng.sample(range(1, n + 1), rng.randint(max(1, n // 4), max(1, n // 2)))))
+ tabu_set.add(remove)
+ tabu.append(remove)
+ if len(tabu) > tabu_size:
+ tabu_set.discard(tabu.popleft())
+
+ remove_set = set(remove)
+ temp = []
+ for r in best:
+ nr = [c for c in r if c not in remove_set]
+ if nr:
+ temp.append(nr)
+ temp = greedy_repair(temp, sorted(remove), inst, rng)
+ temp = local_search(temp, inst)
+ cost = sum(route_dist(r, dist) for r in temp)
+ if cost < best_cost - 1e-9:
+ best = [list(r) for r in temp]
+ best_cost = cost
+ return best
+
+
+def grasp_solve(
+ inst: dict, starts: int = 40, seed: int = 123, lns_iters: int | None = None
+) -> list[list[int]]:
+ """GRASP-style multi-start: randomized savings + local search, keep best.
+
+ When `lns_iters` is given, refine the best solution with deterministic
+ iteration-budgeted LNS.
+ """
+ rng = random.Random(seed)
+ best: list[list[int]] | None = None
+ best_d = float("inf")
+ dist = inst["distance"]
+ for k in range(starts):
+ # First start uses the deterministic savings order (guaranteed baseline);
+ # remaining starts use randomized savings for diversification.
+ routes = savings_solve(inst, rng=None if k == 0 else rng)
+ routes = [two_opt(r, dist) for r in routes]
+ routes = relocate_improve(routes, inst)
+ routes = swap_improve(routes, inst)
+ routes = [two_opt(r, dist) for r in routes]
+ d = sum(route_dist(r, dist) for r in routes)
+ if d < best_d:
+ best_d = d
+ best = routes
+ if best is not None and lns_iters is not None and lns_iters > 0:
+ best = lns_improve(best, inst, random.Random(seed + 1), lns_iters)
+ return best or []
+
+
+def main(argv: list[str] | None = None) -> int:
+ parser = argparse.ArgumentParser(description="Generate CVRP reference distances")
+ parser.add_argument("--starts", type=int, default=40, help="GRASP multi-start count")
+ parser.add_argument(
+ "--iterations",
+ type=int,
+ default=None,
+ help="LNS iterations per instance (default: max(100, 4*n))",
+ )
+ args = parser.parse_args(argv)
+
+ # A few fixed seeds diversify the search; the best distance per instance is
+ # kept. The seed list is a constant so regeneration is byte-identical.
+ seeds = (123, 2024, 7)
+ results = {}
+
+ def _solve_dir(inst_dir: Path) -> None:
+ for path in sorted(inst_dir.glob("*.vrp")):
+ inst = parse_instance(path)
+ iters = args.iterations if args.iterations else max(100, 4 * inst["n"])
+ best_total = min(
+ sum(
+ route_dist(r, inst["distance"])
+ for r in grasp_solve(inst, starts=args.starts, seed=seed, lns_iters=iters)
+ )
+ for seed in seeds
+ )
+ results[inst["name"]] = best_total
+ print(f"{inst['name']}: ref_dist={best_total} (iters={iters}, seeds={seeds})")
+
+ _solve_dir(INST_DIR)
+ if HELDOUT_DIR.is_dir():
+ _solve_dir(HELDOUT_DIR)
+ OUT_JSON.write_text(
+ json.dumps(results, ensure_ascii=False, indent=2, sort_keys=True) + "\n",
+ encoding="utf-8",
+ newline="\n", # force LF so regeneration is byte-identical on any OS
+ )
+ print(f"wrote {OUT_JSON}")
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/benchmarks/VehicleRouting/CVRP/verification/requirements.txt b/benchmarks/VehicleRouting/CVRP/verification/requirements.txt
new file mode 100644
index 00000000..38ddc552
--- /dev/null
+++ b/benchmarks/VehicleRouting/CVRP/verification/requirements.txt
@@ -0,0 +1,2 @@
+# The evaluator uses only the Python standard library.
+# No third-party dependencies are required.
diff --git a/benchmarks/VehicleRouting/CVRP/verification/test_evaluator.py b/benchmarks/VehicleRouting/CVRP/verification/test_evaluator.py
new file mode 100644
index 00000000..13c8fa30
--- /dev/null
+++ b/benchmarks/VehicleRouting/CVRP/verification/test_evaluator.py
@@ -0,0 +1,274 @@
+"""Unit tests for the CVRP evaluator (stdlib unittest, no third-party deps).
+
+Run from the CVRP task directory:
+ python verification/test_evaluator.py
+or from the repo root:
+ python -m unittest discover -s benchmarks/VehicleRouting/CVRP/verification -p 'test_*.py'
+"""
+from __future__ import annotations
+
+import json
+import shutil
+import sys
+import tempfile
+import unittest
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+import evaluator as ev # noqa: E402
+
+CVRP_ROOT = Path(__file__).resolve().parents[1]
+
+
+class TestParseInstance(unittest.TestCase):
+ def test_parse_public_instance(self):
+ inst = ev.parse_instance(CVRP_ROOT / "data/instances" / "VRP-19-2.vrp")
+ self.assertEqual(inst["name"], "VRP-19-2")
+ self.assertEqual(inst["n"], 19) # depot id 1, customers 2..20
+ self.assertEqual(inst["capacity"], 270)
+ self.assertEqual(len(inst["demand"]), 20)
+ self.assertEqual(inst["demand"][0], 0)
+ self.assertEqual(len(inst["distance"]), 20)
+ self.assertEqual(len(inst["distance"][0]), 20)
+ self.assertEqual(inst["distance"][0][0], 0)
+
+ def test_parse_heldout_instance(self):
+ inst = ev.parse_instance(CVRP_ROOT / "data/instances_heldout" / "VHO-22-3.vrp")
+ self.assertEqual(inst["name"], "VHO-22-3")
+ self.assertEqual(inst["n"], 22)
+
+
+class TestRouteDistance(unittest.TestCase):
+ def test_known_distance(self):
+ # 1D line: depot at 0, customers at 3 and 7 => depot->3->7->depot = 17
+ dist = [
+ [0, 3, 7],
+ [3, 0, 4],
+ [7, 4, 0],
+ ]
+ self.assertEqual(ev.route_distance([[1, 2]], dist), 3 + 4 + 7)
+ self.assertEqual(ev.route_distance([], dist), 0)
+
+
+class TestValidate(unittest.TestCase):
+ def setUp(self):
+ self.inst = ev.parse_instance(CVRP_ROOT / "data/instances" / "VRP-19-2.vrp")
+
+ def test_valid_single_customer_routes(self):
+ routes = [[c] for c in range(1, self.inst["n"] + 1)]
+ ok, note, dist = ev.validate(routes, self.inst)
+ self.assertTrue(ok, note)
+ self.assertIsInstance(dist, int)
+ self.assertGreater(dist, 0)
+
+ def test_duplicate_customer(self):
+ routes = [[1, 1]] + [[c] for c in range(2, self.inst["n"] + 1)]
+ ok, note, _ = ev.validate(routes, self.inst)
+ self.assertFalse(ok)
+ self.assertIn("more than once", note)
+
+ def test_capacity_violation(self):
+ routes = [list(range(1, self.inst["n"] + 1))]
+ ok, note, _ = ev.validate(routes, self.inst)
+ self.assertFalse(ok)
+ self.assertIn("capacity", note)
+
+ def test_missing_customer(self):
+ routes = [[1, 2]]
+ ok, note, _ = ev.validate(routes, self.inst)
+ self.assertFalse(ok)
+ self.assertIn("not served", note)
+
+ def test_out_of_range_customer(self):
+ routes = [[self.inst["n"] + 5]] + [[c] for c in range(1, self.inst["n"] + 1)]
+ ok, note, _ = ev.validate(routes, self.inst)
+ self.assertFalse(ok)
+ self.assertIn("out of range", note)
+
+ def test_non_integer_customer(self):
+ routes = [[1.5]] + [[c] for c in range(2, self.inst["n"] + 1)]
+ ok, note, _ = ev.validate(routes, self.inst)
+ self.assertFalse(ok)
+
+
+class TestCheckCandidate(unittest.TestCase):
+ def setUp(self):
+ self.tmp = Path(tempfile.mkdtemp(prefix="cvrp_test_"))
+ self.valid_candidate = CVRP_ROOT / "baseline" / "solver.py"
+
+ def tearDown(self):
+ shutil.rmtree(self.tmp, ignore_errors=True)
+
+ def _write(self, name: str, content: str) -> Path:
+ path = self.tmp / name
+ path.write_text(content, encoding="utf-8")
+ return path
+
+ def test_baseline_is_admissible(self):
+ self.assertEqual(ev.check_candidate(self.valid_candidate), [])
+
+ def test_import_reference_solver_rejected(self):
+ src = self.valid_candidate.read_text(encoding="utf-8")
+ bad = src.replace(
+ "def solve(instance):", "import verification.ref_solver as rs\ndef solve(instance):", 1
+ )
+ path = self._write("bad_import.py", bad)
+ issues = ev.check_candidate(path)
+ self.assertTrue(any("verification" in i or "ref_solver" in i for i in issues))
+
+ def test_hardcoding_rejected(self):
+ src = self.valid_candidate.read_text(encoding="utf-8")
+ bad = src.replace(
+ "def solve(instance):",
+ 'def solve(instance):\n if instance["name"] == "VRP-19-2": return [[1]]',
+ 1,
+ )
+ path = self._write("bad_hardcode.py", bad)
+ issues = ev.check_candidate(path)
+ self.assertTrue(any("hardcode" in i for i in issues))
+
+ def test_missing_markers_rejected(self):
+ src = self.valid_candidate.read_text(encoding="utf-8")
+ bad = src.replace(ev.EVOLVE_START, "").replace(ev.EVOLVE_END, "")
+ path = self._write("bad_markers.py", bad)
+ issues = ev.check_candidate(path)
+ self.assertTrue(any("EVOLVE-BLOCK" in i for i in issues))
+
+ def test_absolute_path_rejected(self):
+ src = self.valid_candidate.read_text(encoding="utf-8")
+ bad = src + "\n# C:\\\\Users\\\\x\\\\reference.json\n"
+ path = self._write("bad_path.py", bad)
+ issues = ev.check_candidate(path)
+ self.assertTrue(any("absolute" in i for i in issues))
+
+
+class TestLoadReference(unittest.TestCase):
+ def test_reference_contains_all_instances(self):
+ ref = ev.load_reference()
+ self.assertEqual(len(ref), 24) # 12 public + 12 held-out
+ for name in (
+ "VRP-19-2",
+ "VRP-60-9",
+ "VHO-22-3",
+ "VHO-58-9",
+ ):
+ self.assertIn(name, ref)
+ self.assertGreater(ref[name], 0)
+
+
+class TestEvaluateBaseline(unittest.TestCase):
+ def test_baseline_scores(self):
+ import os
+
+ os.environ["CVRP_EVAL_INSTANCES"] = "VRP-19-2 VHO-22-3"
+ try:
+ result = ev.evaluate(str(CVRP_ROOT / "baseline" / "solver.py"))
+ finally:
+ os.environ.pop("CVRP_EVAL_INSTANCES", None)
+ metrics = result["metrics"]
+ self.assertEqual(metrics["valid"], 1.0)
+ self.assertEqual(metrics["instances"], 2.0)
+ self.assertGreater(metrics["combined_score"], 40.0)
+ self.assertLess(metrics["combined_score"], 100.0)
+ # artifacts must not leak reference distances
+ artifacts = result["artifacts"]
+ self.assertNotIn("reference", artifacts)
+ self.assertEqual(artifacts["reference_instance_count"], 24.0)
+
+
+class TestGenerateMode(unittest.TestCase):
+ """Runtime instance generation (anti-hardcoding)."""
+
+ def _run(self, seed, count=None, instances=None):
+ import os
+
+ os.environ["CVRP_EVAL_GENERATE_SEED"] = str(seed)
+ if count is not None:
+ os.environ["CVRP_EVAL_GENERATE_COUNT"] = str(count)
+ if instances:
+ os.environ["CVRP_EVAL_INSTANCES"] = instances
+ try:
+ return ev.evaluate(str(CVRP_ROOT / "baseline" / "solver.py"))["metrics"]
+ finally:
+ os.environ.pop("CVRP_EVAL_GENERATE_SEED", None)
+ os.environ.pop("CVRP_EVAL_GENERATE_COUNT", None)
+ os.environ.pop("CVRP_EVAL_INSTANCES", None)
+
+ def test_generates_extra_instances(self):
+ # 2 fixed + 4 generated = 6 instances, all valid.
+ m = self._run(seed=42, count=4, instances="VRP-19-2 VHO-22-3")
+ self.assertEqual(m["valid"], 1.0)
+ self.assertEqual(m["instances"], 6.0)
+ names = set(m["per_instance"].keys())
+ self.assertIn("VRP-19-2", names)
+ self.assertIn("VHO-22-3", names)
+ self.assertTrue(any(n.startswith("GEN-42-") for n in names))
+
+ def test_same_seed_deterministic(self):
+ a = self._run(seed=42, count=3)
+ b = self._run(seed=42, count=3)
+ self.assertEqual(a["combined_score"], b["combined_score"])
+ self.assertEqual(
+ sorted(a["per_instance"].keys()), sorted(b["per_instance"].keys())
+ )
+
+ def test_different_seed_different_instances(self):
+ a = self._run(seed=42, count=3)
+ b = self._run(seed=7, count=3)
+ self.assertNotEqual(
+ sorted(a["per_instance"].keys()), sorted(b["per_instance"].keys())
+ )
+
+ def test_unset_seed_preserves_old_behaviour(self):
+ import os
+
+ os.environ["CVRP_EVAL_INSTANCES"] = "VRP-19-2 VHO-22-3"
+ try:
+ m = ev.evaluate(str(CVRP_ROOT / "baseline" / "solver.py"))["metrics"]
+ finally:
+ os.environ.pop("CVRP_EVAL_INSTANCES", None)
+ self.assertEqual(m["instances"], 2.0)
+ self.assertFalse(any(n.startswith("GEN-") for n in m["per_instance"]))
+
+
+class TestHeldoutFromHost(unittest.TestCase):
+ """Held-out instances are read from the host source dir, not the sandbox copy."""
+
+ def test_heldout_read_from_host_source(self):
+ import os
+ import shutil
+ import tempfile
+
+ host = Path(tempfile.mkdtemp(prefix="cvrp_host_"))
+ heldout_dir = host / "data" / "instances_heldout"
+ heldout_dir.mkdir(parents=True)
+ src_vho = CVRP_ROOT / "data" / "instances_heldout" / "VHO-22-3.vrp"
+ shutil.copy2(src_vho, heldout_dir / "VHO-22-3.vrp")
+ os.environ["FRONTIER_EVAL_UNIFIED_SOURCE_BENCHMARK_DIR"] = str(host)
+ os.environ["CVRP_EVAL_INSTANCES"] = "VRP-19-2 VHO-22-3"
+ try:
+ m = ev.evaluate(str(CVRP_ROOT / "baseline" / "solver.py"))["metrics"]
+ finally:
+ os.environ.pop("FRONTIER_EVAL_UNIFIED_SOURCE_BENCHMARK_DIR", None)
+ os.environ.pop("CVRP_EVAL_INSTANCES", None)
+ shutil.rmtree(host, ignore_errors=True)
+ self.assertEqual(m["valid"], 1.0)
+ self.assertEqual(m["instances"], 2.0)
+ self.assertIn("VRP-19-2", m["per_instance"])
+ self.assertIn("VHO-22-3", m["per_instance"])
+
+
+class TestSplitEvolveBlocks(unittest.TestCase):
+ def test_split(self):
+ src = "a\n# EVOLVE-BLOCK-START\nb\n# EVOLVE-BLOCK-END\nc\n"
+ before, between, after = ev._split_evolve_blocks(src)
+ self.assertEqual(before, "a\n")
+ self.assertIn("b", between)
+ self.assertEqual(after, "\nc\n")
+
+ def test_missing_end(self):
+ self.assertIsNone(ev._split_evolve_blocks("a\n# EVOLVE-BLOCK-START\nb\n"))
+
+
+if __name__ == "__main__":
+ unittest.main(verbosity=2)
diff --git a/benchmarks/VehicleRouting/CVRP/verification/test_frontier_eval_evaluator.py b/benchmarks/VehicleRouting/CVRP/verification/test_frontier_eval_evaluator.py
new file mode 100644
index 00000000..a96bb097
--- /dev/null
+++ b/benchmarks/VehicleRouting/CVRP/verification/test_frontier_eval_evaluator.py
@@ -0,0 +1,122 @@
+"""Unit tests for the sandbox evaluator (frontier_eval/evaluator.py).
+
+The sandbox evaluator is a self-contained copy of the verification evaluator
+(parsing / validation / scoring / integrity checks embedded, since no
+`verification/` files are copied into the sandbox). These tests exercise it
+directly so the two copies cannot silently drift apart.
+
+Run from the CVRP task directory:
+ python verification/test_frontier_eval_evaluator.py
+"""
+from __future__ import annotations
+
+import importlib.util
+import json
+import os
+import shutil
+import sys
+import tempfile
+import unittest
+from pathlib import Path
+
+CVRP_ROOT = Path(__file__).resolve().parents[1]
+SANDBOX_EVALUATOR = CVRP_ROOT / "frontier_eval" / "evaluator.py"
+
+_spec = importlib.util.spec_from_file_location("_sandbox_evaluator", SANDBOX_EVALUATOR)
+_sandbox = importlib.util.module_from_spec(_spec)
+assert _spec.loader is not None
+_spec.loader.exec_module(_sandbox)
+
+BASELINE = CVRP_ROOT / "baseline" / "solver.py"
+
+
+class TestSandboxBaseline(unittest.TestCase):
+ def test_baseline_scores_and_no_leak(self):
+ m = _sandbox.evaluate(str(BASELINE))["metrics"]
+ self.assertEqual(m["valid"], 1.0)
+ self.assertEqual(m["instances"], 24.0)
+ self.assertTrue(all(n.startswith(("VRP-", "VHO-")) for n in m["per_instance"]))
+ self.assertGreater(m["combined_score"], 40.0)
+ self.assertLess(m["combined_score"], 100.0)
+
+ def test_artifacts_do_not_leak_reference(self):
+ result = _sandbox.evaluate(str(BASELINE))
+ self.assertNotIn("reference", result["artifacts"])
+ self.assertEqual(result["artifacts"]["reference_instance_count"], 24.0)
+
+
+class TestSandboxHeldoutFromHost(unittest.TestCase):
+ def test_heldout_read_from_host_source(self):
+ host = Path(tempfile.mkdtemp(prefix="cvrp_sb_host_"))
+ heldout_dir = host / "data" / "instances_heldout"
+ heldout_dir.mkdir(parents=True)
+ shutil.copy2(
+ CVRP_ROOT / "data" / "instances_heldout" / "VHO-22-3.vrp",
+ heldout_dir / "VHO-22-3.vrp",
+ )
+ os.environ["FRONTIER_EVAL_UNIFIED_SOURCE_BENCHMARK_DIR"] = str(host)
+ os.environ["CVRP_EVAL_INSTANCES"] = "VRP-19-2 VHO-22-3"
+ try:
+ m = _sandbox.evaluate(str(BASELINE))["metrics"]
+ finally:
+ os.environ.pop("FRONTIER_EVAL_UNIFIED_SOURCE_BENCHMARK_DIR", None)
+ os.environ.pop("CVRP_EVAL_INSTANCES", None)
+ shutil.rmtree(host, ignore_errors=True)
+ self.assertEqual(m["valid"], 1.0)
+ self.assertEqual(m["instances"], 2.0)
+ self.assertIn("VHO-22-3", m["per_instance"])
+
+
+class TestSandboxGeneration(unittest.TestCase):
+ def test_generate_mode(self):
+ # The sandbox evaluator generates instances by loading the host
+ # generator/reference solver, so it needs the host source dir.
+ os.environ["FRONTIER_EVAL_UNIFIED_SOURCE_BENCHMARK_DIR"] = str(CVRP_ROOT)
+ os.environ["CVRP_EVAL_GENERATE_SEED"] = "42"
+ os.environ["CVRP_EVAL_GENERATE_COUNT"] = "4"
+ try:
+ m = _sandbox.evaluate(str(BASELINE))["metrics"]
+ finally:
+ os.environ.pop("FRONTIER_EVAL_UNIFIED_SOURCE_BENCHMARK_DIR", None)
+ os.environ.pop("CVRP_EVAL_GENERATE_SEED", None)
+ os.environ.pop("CVRP_EVAL_GENERATE_COUNT", None)
+ self.assertEqual(m["valid"], 1.0)
+ self.assertEqual(m["instances"], 28.0)
+ self.assertTrue(any(n.startswith("GEN-42-") for n in m["per_instance"]))
+
+
+class TestSandboxPreflight(unittest.TestCase):
+ def test_cheating_candidate_rejected(self):
+ cheat = CVRP_ROOT / "baseline" / "solver.py"
+ src = cheat.read_text(encoding="utf-8")
+ start = src.find("# EVOLVE-BLOCK-START")
+ end = src.find("# EVOLVE-BLOCK-END")
+ injected = (
+ src[: start + len("# EVOLVE-BLOCK-START")]
+ + "\n import ref_solver # forbidden\n return [[c] for c in range(1, instance['n'] + 1)]\n"
+ + src[end:]
+ )
+ tmp = Path(tempfile.mkdtemp(prefix="cvrp_sb_cheat_"))
+ cand = tmp / "cheat.py"
+ cand.write_text(injected, encoding="utf-8")
+ try:
+ m = _sandbox.evaluate(str(cand))["metrics"]
+ finally:
+ shutil.rmtree(tmp, ignore_errors=True)
+ self.assertEqual(m["valid"], 0.0)
+ self.assertEqual(m["combined_score"], 0.0)
+
+
+class TestSandboxConsistency(unittest.TestCase):
+ def test_parse_instance_matches_verification_copy(self):
+ sys.path.insert(0, str(Path(__file__).resolve().parent))
+ import evaluator as verif # noqa: E402
+
+ inst = CVRP_ROOT / "data" / "instances" / "VRP-19-2.vrp"
+ a = verif.parse_instance(inst)
+ b = _sandbox.parse_instance(inst)
+ self.assertEqual(a, b)
+
+
+if __name__ == "__main__":
+ unittest.main(verbosity=2)
diff --git a/benchmarks/VehicleRouting/CVRP/verification/test_ref_solver.py b/benchmarks/VehicleRouting/CVRP/verification/test_ref_solver.py
new file mode 100644
index 00000000..1a975d54
--- /dev/null
+++ b/benchmarks/VehicleRouting/CVRP/verification/test_ref_solver.py
@@ -0,0 +1,83 @@
+"""Unit tests for verification/ref_solver.py (stdlib unittest, no third-party deps).
+
+Run from the CVRP task directory:
+ python verification/test_ref_solver.py
+"""
+from __future__ import annotations
+
+import json
+import sys
+import unittest
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+import ref_solver as rs # noqa: E402
+
+CVRP_ROOT = Path(__file__).resolve().parents[1]
+INST = CVRP_ROOT / "data" / "instances" / "VRP-19-2.vrp"
+
+
+class TestGraspDeterministic(unittest.TestCase):
+ def test_same_seed_same_distance(self):
+ inst = rs.parse_instance(INST)
+ args = dict(starts=3, seed=7, lns_iters=5)
+ d1 = sum(rs.route_dist(r, inst["distance"]) for r in rs.grasp_solve(inst, **args))
+ d2 = sum(rs.route_dist(r, inst["distance"]) for r in rs.grasp_solve(inst, **args))
+ self.assertEqual(d1, d2)
+
+ def test_different_seeds_usually_differ(self):
+ inst = rs.parse_instance(INST)
+ d_a = sum(
+ rs.route_dist(r, inst["distance"])
+ for r in rs.grasp_solve(inst, starts=3, seed=1, lns_iters=5)
+ )
+ d_b = sum(
+ rs.route_dist(r, inst["distance"])
+ for r in rs.grasp_solve(inst, starts=3, seed=999, lns_iters=5)
+ )
+ # The GRASP perturbation makes different seeds explore different
+ # neighbourhoods; identical distance is possible but unlikely.
+ self.assertIsInstance(d_a, int)
+ self.assertIsInstance(d_b, int)
+
+
+class TestGraspFeasible(unittest.TestCase):
+ def test_full_coverage_and_capacity(self):
+ inst = rs.parse_instance(INST)
+ routes = rs.grasp_solve(inst, starts=3, seed=7, lns_iters=5)
+ covered = [c for route in routes for c in route]
+ self.assertEqual(sorted(covered), list(range(1, inst["n"] + 1)))
+ for route in routes:
+ load = sum(inst["demand"][c] for c in route)
+ self.assertLessEqual(load, inst["capacity"])
+
+ def test_returns_list_of_lists(self):
+ inst = rs.parse_instance(INST)
+ routes = rs.grasp_solve(inst, starts=3, seed=7, lns_iters=5)
+ self.assertIsInstance(routes, list)
+ for route in routes:
+ self.assertIsInstance(route, list)
+
+
+class TestReferenceJsonConsistency(unittest.TestCase):
+ def test_matches_reference_json(self):
+ """Reproduce main()'s computation for the smallest instance and check
+ it equals the checked-in reference distance (byte-identical, fixed
+ seed list (123, 2024, 7))."""
+ inst = rs.parse_instance(INST)
+ iters = max(100, 4 * inst["n"])
+ best = min(
+ sum(
+ rs.route_dist(r, inst["distance"])
+ for r in rs.grasp_solve(inst, starts=40, seed=seed, lns_iters=iters)
+ )
+ for seed in (123, 2024, 7)
+ )
+ ref = json.loads(
+ (CVRP_ROOT / "data" / "reference.json").read_text(encoding="utf-8")
+ )
+ self.assertEqual(best, ref[inst["name"]])
+
+
+if __name__ == "__main__":
+ unittest.main(verbosity=2)
diff --git a/benchmarks/VehicleRouting/CVRP/verification/test_validator.py b/benchmarks/VehicleRouting/CVRP/verification/test_validator.py
new file mode 100644
index 00000000..7146b929
--- /dev/null
+++ b/benchmarks/VehicleRouting/CVRP/verification/test_validator.py
@@ -0,0 +1,205 @@
+"""Unit tests for verification/validator.py (stdlib unittest, no third-party deps).
+
+Run from the CVRP task directory:
+ python verification/test_validator.py
+"""
+from __future__ import annotations
+
+import shutil
+import sys
+import tempfile
+import unittest
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+import validator as vd # noqa: E402
+import evaluator as ev # noqa: E402 (for parse_instance in probe selection)
+
+CVRP_ROOT = Path(__file__).resolve().parents[1]
+BASELINE = CVRP_ROOT / "baseline" / "solver.py"
+BASELINE_SRC = BASELINE.read_text(encoding="utf-8")
+INSTANCE_PATHS = sorted((CVRP_ROOT / "data" / "instances").glob("*.vrp")) + sorted(
+ (CVRP_ROOT / "data" / "instances_heldout").glob("*.vrp")
+)
+
+
+class TestStaticChecks(unittest.TestCase):
+ def setUp(self):
+ self.tmp = Path(tempfile.mkdtemp(prefix="cvrp_val_"))
+
+ def tearDown(self):
+ shutil.rmtree(self.tmp, ignore_errors=True)
+
+ def _check(self, src: str) -> list[str]:
+ path = self.tmp / "candidate.py"
+ path.write_text(src, encoding="utf-8")
+ return vd.check_candidate(path, baseline_path=BASELINE)
+
+ def test_baseline_is_clean(self):
+ self.assertEqual(vd.check_candidate(BASELINE, baseline_path=BASELINE), [])
+
+ def test_missing_markers(self):
+ src = BASELINE_SRC.replace(vd.EVOLVE_START, "").replace(vd.EVOLVE_END, "")
+ issues = self._check(src)
+ self.assertTrue(any("EVOLVE-BLOCK" in i for i in issues))
+
+ def test_outside_block_changed(self):
+ src = BASELINE_SRC.replace(
+ '"""CVRP candidate solver', '"""HACKED candidate solver', 1
+ )
+ issues = self._check(src)
+ self.assertTrue(any("outside EVOLVE-BLOCK" in i for i in issues))
+
+ def test_import_reference_solver(self):
+ src = BASELINE_SRC.replace(
+ "def solve(instance):",
+ "import verification.ref_solver as rs\ndef solve(instance):",
+ 1,
+ )
+ issues = self._check(src)
+ self.assertTrue(any("ref_solver" in i for i in issues))
+
+ def test_read_reference_json(self):
+ src = BASELINE_SRC.replace(
+ "def solve(instance):",
+ 'def solve(instance):\n _ = open("data/reference.json").read()',
+ 1,
+ )
+ issues = self._check(src)
+ self.assertTrue(any("reference.json" in i for i in issues))
+
+ def test_hardcode_dict_key(self):
+ src = BASELINE_SRC.replace(
+ "def solve(instance):",
+ 'def solve(instance):\n return {"VRP-19-2": [[1]]}[instance["name"]]',
+ 1,
+ )
+ issues = self._check(src)
+ self.assertTrue(any("hardcode" in i for i in issues))
+
+ def test_hardcode_equality(self):
+ src = BASELINE_SRC.replace(
+ "def solve(instance):",
+ 'def solve(instance):\n if instance["name"] == "VHO-22-3":\n'
+ ' return [[1]]',
+ 1,
+ )
+ issues = self._check(src)
+ self.assertTrue(any("hardcode" in i for i in issues))
+
+ def test_absolute_path(self):
+ src = BASELINE_SRC + '\n# C:\\\\Users\\\\x\\\\reference.json\n'
+ issues = self._check(src)
+ self.assertTrue(any("absolute" in i for i in issues))
+
+ def test_static_check_direct(self):
+ # static_check_source with no baseline skips the fixed-region diff.
+ issues = vd.static_check_source(BASELINE_SRC, baseline_src=None)
+ self.assertEqual(issues, [])
+
+ def test_comment_verification_not_rejected(self):
+ # "verification" as a bare word (e.g. in a comment) is NOT a violation;
+ # only import/from/module-path usage is.
+ src = BASELINE_SRC.replace(
+ "def solve(instance):",
+ "# verification pass for small instances\ndef solve(instance):",
+ 1,
+ )
+ issues = self._check(src)
+ self.assertFalse(any("verification" in i for i in issues))
+
+ def test_import_verification_rejected(self):
+ src = BASELINE_SRC.replace(
+ "def solve(instance):",
+ "import verification\nfrom verification import evaluator\n"
+ "def solve(instance):",
+ 1,
+ )
+ issues = self._check(src)
+ self.assertTrue(any("verification" in i for i in issues))
+
+
+class TestDeterminism(unittest.TestCase):
+ def setUp(self):
+ self.tmp = Path(tempfile.mkdtemp(prefix="cvrp_det_test_"))
+ self.inst_path = CVRP_ROOT / "data" / "instances" / "VRP-19-2.vrp"
+ import sys as _sys
+
+ self.python = _sys.executable
+
+ def tearDown(self):
+ shutil.rmtree(self.tmp, ignore_errors=True)
+
+ def test_deterministic_baseline_passes(self):
+ ok, note = vd.check_determinism(
+ self.python, BASELINE, self.inst_path, timeout=60
+ )
+ self.assertTrue(ok, note)
+
+ def test_nondeterministic_candidate_fails(self):
+ src = BASELINE_SRC.replace(
+ "def solve(instance):",
+ "def solve(instance):\n import random\n"
+ " return [[random.randint(1, instance['n'])\n"
+ " for _ in range(instance['n'])]]",
+ 1,
+ )
+ path = self.tmp / "nondet.py"
+ path.write_text(src, encoding="utf-8")
+ ok, note = vd.check_determinism(self.python, path, self.inst_path, timeout=60)
+ self.assertFalse(ok)
+ self.assertIn("not deterministic", note)
+
+ def test_size_varied_randomness_detected(self):
+ # Deterministic on small instances, random on large ones: the probe
+ # selection must include a large instance, so this is caught.
+ src = BASELINE_SRC.replace(
+ "def solve(instance):",
+ "def solve(instance):\n import random\n"
+ " if instance['n'] <= 30:\n"
+ " return [[c] for c in range(1, instance['n'] + 1)]\n"
+ " return [[random.randint(1, instance['n'])\n"
+ " for _ in range(instance['n'])]]",
+ 1,
+ )
+ path = self.tmp / "sizevar.py"
+ path.write_text(src, encoding="utf-8")
+
+ probes = vd.select_determinism_probes(INSTANCE_PATHS, ev.parse_instance)
+ self.assertEqual(len(probes), 3)
+ # The largest probe has n > 30, so the candidate is random there.
+ ok, note = vd.check_determinism(self.python, path, probes[-1], timeout=60)
+ self.assertFalse(ok)
+ self.assertIn("not deterministic", note)
+
+ def test_probe_selection_spans_sizes(self):
+ probes = vd.select_determinism_probes(INSTANCE_PATHS, ev.parse_instance)
+ ns = sorted(ev.parse_instance(p)["n"] for p in probes)
+ self.assertEqual(len(ns), 3)
+ # min < median < max over the full set (24 instances).
+ all_ns = sorted(ev.parse_instance(p)["n"] for p in INSTANCE_PATHS)
+ self.assertEqual(ns[0], all_ns[0])
+ self.assertEqual(ns[-1], all_ns[-1])
+
+
+class TestCandidateEnvStripsFrontier(unittest.TestCase):
+ """candidate_env must strip FRONTIER_ENGINEERING_ROOT (repo-root side channel)."""
+
+ def test_strips_frontier_engineering_root(self):
+ import os
+
+ import validator as v
+
+ os.environ["FRONTIER_ENGINEERING_ROOT"] = "C:/fake/repo"
+ os.environ["FRONTIER_EVAL_UNIFIED_SOURCE_BENCHMARK_DIR"] = "C:/fake/bench"
+ try:
+ env = v.candidate_env()
+ finally:
+ os.environ.pop("FRONTIER_ENGINEERING_ROOT", None)
+ os.environ.pop("FRONTIER_EVAL_UNIFIED_SOURCE_BENCHMARK_DIR", None)
+ self.assertNotIn("FRONTIER_ENGINEERING_ROOT", env)
+ self.assertNotIn("FRONTIER_EVAL_UNIFIED_SOURCE_BENCHMARK_DIR", env)
+
+
+if __name__ == "__main__":
+ unittest.main(verbosity=2)
diff --git a/benchmarks/VehicleRouting/CVRP/verification/validator.py b/benchmarks/VehicleRouting/CVRP/verification/validator.py
new file mode 100644
index 00000000..421ff527
--- /dev/null
+++ b/benchmarks/VehicleRouting/CVRP/verification/validator.py
@@ -0,0 +1,204 @@
+"""Candidate integrity validator for the CVRP benchmark.
+
+Enforces the task constraints as *executable* checks (not just natural
+language). The evaluator runs these before scoring; any violation marks the
+candidate invalid.
+
+Checks implemented:
+ 1. EVOLVE-BLOCK integrity: markers must exist, and the code outside the
+ EVOLVE-BLOCK region must match the initial baseline byte-for-byte.
+ 2. Forbidden references: the source must not reference the verification
+ module, the reference solver, `reference.json`, or helper internals.
+ 3. Absolute paths: the source must not contain machine-local filesystem
+ paths.
+ 4. Per-instance hardcoding: the source must not embed instance names in a
+ solution-dispatch form (e.g. `"VRP-19-2": [...]`).
+ 5. Determinism: running the candidate twice on the same instance must yield
+ byte-identical output (a scorer needs reproducible results).
+
+Pure standard library. Used by `verification/evaluator.py` and
+`frontier_eval/evaluator.py` (which embeds an equivalent copy for sandbox use).
+"""
+from __future__ import annotations
+
+import json
+import os
+import re
+import subprocess
+import tempfile
+from pathlib import Path
+from typing import Any
+
+EVOLVE_START = "# EVOLVE-BLOCK-START"
+EVOLVE_END = "# EVOLVE-BLOCK-END"
+
+# Strong tokens: almost never appear in legitimate solver code, so a bare
+# substring match is fine (e.g. "ref_solver", "reference.json").
+STRONG_TOKENS = (
+ "ref_solver",
+ "grasp_solve",
+ "savings_solve",
+ "reference.json",
+)
+# Weak token "verification": only rejected when it appears in an
+# import/from/module-path context, so a comment like "verification pass"
+# is NOT a false positive.
+FORBIDDEN_RE = (
+ re.compile(r"\b(?:import|from)\s+verification\b"),
+ re.compile(r"verification[\\/.]"),
+)
+
+# e.g. `"VRP-19-2": [...]` or `"VHO-22-3":` (a dispatch table keyed by name).
+HARDCODE_RE = re.compile(r"[\"'][A-Z][A-Z0-9-]*\d+-\d+[\"']\s*:")
+# Windows drive / POSIX home absolute paths.
+ABS_PATH_RE = re.compile(r"[A-Za-z]:[\\/]|/home/|/Users/")
+
+
+def split_evolve_blocks(src: str) -> tuple[str, str, str] | None:
+ """Return (before_start, between, after_end) or None if markers missing."""
+ start = src.find(EVOLVE_START)
+ end = src.find(EVOLVE_END)
+ if start == -1 or end == -1 or end <= start:
+ return None
+ return (
+ src[:start],
+ src[start + len(EVOLVE_START) : end],
+ src[end + len(EVOLVE_END) :],
+ )
+
+
+def fixed_region(parts: tuple[str, str, str]) -> str:
+ """Everything outside the EVOLVE-BLOCK (the read-only part)."""
+ return parts[0] + parts[2]
+
+
+def static_check_source(src: str, baseline_src: str | None = None) -> list[str]:
+ """Static checks on candidate source text. Returns a list of violations."""
+ issues: list[str] = []
+
+ parts = split_evolve_blocks(src)
+ if parts is None:
+ issues.append("missing EVOLVE-BLOCK-START / EVOLVE-BLOCK-END markers")
+ elif baseline_src is not None:
+ init_parts = split_evolve_blocks(baseline_src)
+ if init_parts is not None and fixed_region(init_parts) != fixed_region(parts):
+ issues.append("code outside EVOLVE-BLOCK differs from initial baseline")
+
+ for token in STRONG_TOKENS:
+ if token in src:
+ issues.append(f"candidate references forbidden token {token!r}")
+ for pat in FORBIDDEN_RE:
+ if pat.search(src):
+ issues.append("candidate references forbidden token 'verification'")
+
+ if ABS_PATH_RE.search(src):
+ issues.append("candidate contains an absolute filesystem path")
+
+ if HARDCODE_RE.search(src):
+ issues.append("candidate hardcodes per-instance solutions by name")
+
+ return issues
+
+
+def check_candidate(
+ solver_path: Path | str, baseline_path: Path | str | None = None
+) -> list[str]:
+ """Read candidate source and run the static checks.
+
+ `baseline_path` is the pristine initial program used to verify the fixed
+ (non-EVOLVE-BLOCK) region. When None, the EVOLVE-BLOCK marker check still
+ applies but the fixed-region diff is skipped.
+ """
+ solver_path = Path(solver_path)
+ try:
+ src = solver_path.read_text(encoding="utf-8", errors="replace")
+ except Exception as exc:
+ return [f"cannot read candidate source: {exc}"]
+
+ baseline_src = None
+ if baseline_path is not None:
+ try:
+ baseline_src = Path(baseline_path).read_text(
+ encoding="utf-8", errors="replace"
+ )
+ except Exception:
+ baseline_src = None
+
+ return static_check_source(src, baseline_src)
+
+
+def candidate_env() -> dict[str, str]:
+ """Environment for the candidate subprocess, stripped of host paths and any
+ reference-distance settings so the candidate cannot locate the scoring
+ baseline on the host. All `FRONTIER_*` variables are removed (the unified
+ runtime sets `FRONTIER_ENGINEERING_ROOT` to the repo root, which would
+ otherwise let a candidate find and import `verification/ref_solver.py` on
+ the host)."""
+ env = os.environ.copy()
+ for key in list(env):
+ upper = key.upper()
+ if upper.startswith("FRONTIER") or key in (
+ "CVRP_EVAL_REFERENCE_JSON",
+ "CVRP_EVAL_REFERENCES",
+ ):
+ del env[key]
+ return env
+
+
+def check_determinism(
+ python: str,
+ solver_path: Path | str,
+ inst_path: Path | str,
+ timeout: float,
+ env: dict[str, str] | None = None,
+) -> tuple[bool, str]:
+ """Run the candidate twice on the same instance; outputs must match.
+
+ Returns (ok, note). A non-deterministic candidate is not reliably
+ scoreable, so it is treated as a violation.
+ """
+ solver_path = Path(solver_path)
+ inst_path = Path(inst_path)
+ outputs: list[Any] = []
+ for _ in range(2):
+ with tempfile.TemporaryDirectory(prefix="cvrp_det_") as td:
+ out_path = Path(td) / "out.json"
+ try:
+ proc = subprocess.run(
+ [python, str(solver_path), str(inst_path), str(out_path)],
+ capture_output=True,
+ text=True,
+ timeout=timeout,
+ cwd=str(solver_path.parent),
+ env=env if env is not None else candidate_env(),
+ )
+ except subprocess.TimeoutExpired:
+ return False, "timeout during determinism check"
+ if proc.returncode != 0:
+ return (
+ False,
+ f"candidate exited with code {proc.returncode} during "
+ f"determinism check: {(proc.stderr or '')[:200]}",
+ )
+ try:
+ outputs.append(json.loads(out_path.read_text(encoding="utf-8")))
+ except Exception as exc:
+ return False, f"cannot parse determinism output: {exc}"
+ if outputs[0] != outputs[1]:
+ return False, "candidate is not deterministic (output differs across two runs)"
+ return True, ""
+
+
+def select_determinism_probes(
+ inst_paths: list, parse_instance, count: int = 3
+) -> list:
+ """Pick min / median / max instances (by customer count) as determinism
+ probes, so a solver that is deterministic on small instances but random on
+ large ones cannot slip through a single-probe check."""
+ parsed = sorted(
+ ((p, parse_instance(p)) for p in inst_paths), key=lambda t: t[1]["n"]
+ )
+ if not parsed:
+ return []
+ idxs = sorted({0, len(parsed) // 2, len(parsed) - 1})
+ return [parsed[i][0] for i in idxs[:count]]
diff --git a/benchmarks/VehicleRouting/README.md b/benchmarks/VehicleRouting/README.md
new file mode 100644
index 00000000..618fdb2e
--- /dev/null
+++ b/benchmarks/VehicleRouting/README.md
@@ -0,0 +1,11 @@
+# Vehicle Routing Domain
+
+Vehicle routing problems (VRP) model logistics and delivery planning: how a
+fleet of vehicles should serve a set of customers from one or more depots,
+subject to capacity, time-window, and other constraints, minimizing cost.
+
+## Tasks
+
+| Task | Problem | Description |
+|------|---------|-------------|
+| [CVRP](CVRP/README.md) | Capacitated VRP | Single depot, identical vehicles with capacity, every customer served once, minimize total distance. |