Test sweep: the offline scenarios must actually run — one was failing silently

[neosun100]

What the sweep did

A full test sweep — unit + regression + property-at-5000-examples + E2E — found the suite green at 3668 and every make target green, then went one layer further and executed the scenarios/ scripts themselves. Nothing covered that layer.

Five scenarios had no test coverage of any kind, and one of them had been failing.

The finding

scenario_agent_authored_loop.py exited 1 while still writing its evidence/*.json, so a committed artifact that claims to prove the happy promotion path actually recorded it being refused:

happy_promotion  ok=False promoted=False refused=1
reason: "safety data missing: dimension(s) ['groundedness'] were never scored
         — a judge's silence is not a pass (fail-closed)"

Not a code defect. INV-PROMOTE-3 is right to refuse it. The scenario's _passing_eval stub scored correctness and safety but never groundedness, and went stale when M18.1 added that requirement. Confirmed pre-existing by re-running on f5daa30 (round 20) — identical failure, so it is not fallout from round 21.

Nothing caught it because nothing ran the scenario. make test covers tests/; the scenarios are exercised only by hand. That is the INV-AUDITMAP shape one level over: an artifact asserting a claim, with no mechanism checking the assertion still holds.

The fix

tests/test_scenarios_execute.py. Every scenario is classified by running it — an earlier pass of this sweep grepped for the word "offline" and mis-classified three live scenarios because their prose mentions offline mode, which is recorded in the module:

13 offline-runnable	executed in CI with AWS_*/SENTINEL_* stripped from the environment, exit 0 asserted
10 live-AWS	exempted, each naming the precondition it needs

An unclassified scenario now fails the build, so a new script is either hermetic or a decision someone recorded.

The agent_authored_loop assertion reads the evidence file — the artifact a reader trusts — and also asserts the three NEGATIVE paths still refuse, so a stub change that made everything pass would not satisfy it. Positive-controlled: reintroducing the stale stub fails both the run and the evidence check.

A second stale artifact, caught by the same mechanism

evidence/feedback_loop_result.json was stale in the same way but with exit 0, so even less visible. Its old sigma_filter_yaml was:

dst_domain|endswith: 'assets.example.com'    # matches evilassets.example.com

No dot boundary — that filter would suppress an attacker-controlled lookalike as "known good". The code was fixed to emit an apex + .suffix pair; the evidence was never regenerated, so the committed artifact still showed the vulnerable output. That scenario is in the offline set now, so it stays current.

Verification

• 3693 passed / 6 skipped (was 3668) — green under random order
• 42 property assertions at max_examples=5000 — a confirmed negative, not an unsearched one
• make smoke (14 passed, zero AWS), make demo, make seed-registry (dual-gate ok, 19 live, web_search correctly pending), make create-harnesses (DRY_RUN) — all exit 0
• ruff clean · both mypy gates clean · make ci green (lint / coverage ≥88 / iac synth / secret-scan)