Evaluation integrity: in-process candidate execution allows score manipulation (reward hacking) in Python-solver tasks

[solarknight]

Summary

While reviewing the evaluation harness, we found that tasks whose candidate is a
Python module loaded in-process (e.g. QuantumComputing/task_02_clifford_t_synthesis)
let a candidate manipulate its own score, because the candidate runs in the same
Python interpreter as the scorer. This bypasses the benchmark's stated integrity
guarantees and makes the resulting score untrustworthy. It appears to be a class-level
issue affecting every task that imports the candidate in-process, not a single task.

Observed on main. We did not verify the frozen v1-arxiv branch.

Root cause

Three independent weaknesses combine in the affected tasks:

1. No process isolation between candidate and scorer.
   verification/utils.py::load_solver() loads the candidate via
   importlib.util.spec_from_file_location(...) + spec.loader.exec_module(module),
   so the candidate's module-level code executes inside the evaluator process.

2. Scoring functions are reachable and reassignable at runtime.
   verification/evaluate.py runs as __main__, and evaluate_case() calls
   transpile_to_clifford_t(...) via module-global name lookup. A candidate can
   reassign that global from within its own import, e.g.:

   # inside candidate solve.py, at import time
   import sys
   sys.modules["__main__"].transpile_to_clifford_t = my_hijacked_version
   
   Because load_solver() is called before the transpile calls in evaluate_case(),
   the replacement is in effect when the score is computed. This lets the candidate
   control both the candidate cost and the reference opt0/opt3 costs — i.e. both
   the numerator and denominator of the normalized score.

3. No correctness / equivalence validation.
   The candidate output is never checked for correctness (no unitary-equivalence check;
   _strip_non_unitary_ops only removes barriers/measures). An empty or arbitrary
   circuit is scored as if it were a valid solution.

Why existing safeguards do not catch it

• The read-only fingerprint check in frontier_eval/tasks/unified/evaluator/python.py
  (_check_readonly_violations) compares on-disk SHA-256 hashes. A runtime monkeypatch
  mutates in-memory objects only; no file changes on disk, so the check passes.
• Filesystem/OS sandboxing (temp-dir copy, and the Docker isolation with
  --cap-drop ALL / --network none / --read-only) isolates the filesystem and OS,
  but the candidate and scorer still share one interpreter, so in-process function
  replacement is unaffected.