Test sweep: the offline scenarios must actually run — one was failing silently

README.md

@@ -13,7 +13,7 @@
   <img alt="python" src="https://img.shields.io/badge/python-3.10%2B-2997ff"/>
   <img alt="bedrock-agentcore" src="https://img.shields.io/badge/Amazon%20Bedrock-AgentCore%20Harness-ff9900"/>
   <img alt="version" src="https://img.shields.io/badge/version-0.4.0-2997ff"/>
-  <img alt="tests" src="https://img.shields.io/badge/offline%20tests-3668%20passing-1D8102"/>
+  <img alt="tests" src="https://img.shields.io/badge/offline%20tests-3693%20passing-1D8102"/>
   <img alt="coverage" src="https://img.shields.io/badge/coverage-90%25-1D8102"/>
   <img alt="milestones" src="https://img.shields.io/badge/milestones-M0--M15%20delivered-1D8102"/>
   <img alt="hardening" src="https://img.shields.io/badge/adversarial%20audit-100%20defects%20fixed-8b5cf6"/>
@@ -99,7 +99,7 @@ Honest build status per capability — mirrors the self-audit.
 | **Tools** | `nvd_lookup` / `epss_kev` / `attack_lookup` / `web_search` | 🟡 **reference stubs** (offline-safe, contract-tested) | `tools/`, `tests/test_tool_handlers.py` |
 | **Tools** | `siem_query` / `asset_lookup` / `enrich_ioc` / `ops_query` — backend-pluggable | 🟢 **built + tested** (offline mock default; `*_LIVE`=1 switches to a real stdlib-HTTP client — env-driven URL + bearer, timeouts, all failures→`upstream_error` with no silent fallback — proven end-to-end against an in-process 127.0.0.1 mock server, zero external network) | `tools/{siem_query,asset_lookup,enrich_ioc,ops_query}/`, `tests/test_*_live.py` |
 
-🟢 built & validated · 🟡 built, partial · 🟠 designed with loadable config · ⚪ design narrative only. **3668 offline tests pass** (+6 skipped when optional deps absent).
+🟢 built & validated · 🟡 built, partial · 🟠 designed with loadable config · ⚪ design narrative only. **3693 offline tests pass** (+6 skipped when optional deps absent).
 
 ## 🚀 Quickstart
 
@@ -247,7 +247,7 @@ Borrowed patterns (see [`docs/BLUEPRINT.md`](docs/BLUEPRINT.md)): supervisor→s
 | [`docs/GOVERNANCE.md`](docs/GOVERNANCE.md) | Registry dual-gate, HITL, sandbox hooks, and tag-guard controls |
 | [`docs/COMPLIANCE.md`](docs/COMPLIANCE.md) | Capability → SOC 2 / ISO 27001 / NIST CSF 2.0 control mapping (anchors machine-verified) |
 | [`docs/OBSERVABILITY.md`](docs/OBSERVABILITY.md) | Logging (`logutil`), metrics (token/latency/tool-call/error/eval), the OTEL/Transaction-Search path |
-| [`docs/TESTING.md`](docs/TESTING.md) | The 3668-test offline suite: layout, determinism, how to run |
+| [`docs/TESTING.md`](docs/TESTING.md) | The 3693-test offline suite: layout, determinism, how to run |
 | [`docs/FIDELITY-REPORT.md`](docs/FIDELITY-REPORT.md) | The self-audit — real vs. built vs. designed, with limits stated |
 | [`docs/ROADMAP.md`](docs/ROADMAP.md) | Delivered milestones (M0–M12) and what's next |
 | [**API reference (live)**](https://aws-samples.github.io/sample-sentinel-harness/) | Rendered `sentinel_harness` API docs (pdoc → GitHub Pages) |
@@ -285,7 +285,7 @@ sentinel-harness/
 ├── iac-cdk/              L3 CDK stacks (9; guardrail/identity/obs/vpc live) 🟢
 ├── iac-terraform/        deployable Terraform mirror (validate-clean)  🟢
 ├── docs/                 QUICKSTART · ARCHITECTURE · BLUEPRINT · SETUP · HARNESSES · GOVERNANCE · TESTING · FIDELITY-REPORT · ROADMAP
-├── tests/                offline unit + config tests (3668)     🟢
+├── tests/                offline unit + config tests (3693)     🟢
 └── .github/workflows/    CI incl. a customer-name / secret gate
 ```
 

docs/FIDELITY-REPORT.md

@@ -29,7 +29,7 @@ This is real, not narration:
 - **Layer 3 foundation ships runnable, tested code.** The dual-gate tool/skill registry, PreToolUse sandbox hook, and Agent Factory (fleet provision, dry-run, cross-env tag-guard) are built and unit-tested.
 - **Native IaC, not hand-rolled resources.** The gateway / registry / memory / harness stacks use the native `AWS::BedrockAgentCore::*` CloudFormation types. Per the README status matrix, the **Gateway and Memory CFN types are registered**; the Registry type is not yet in CFN (see limitations).
 - **Config path works.** `pip install -e .` succeeds (`[tool.setuptools] packages = ["sentinel_harness", "intake"]`), the `sentinel` console script works, and `sentinel create <harness.yaml>` loads real config via `sentinel_harness/loader.py` (systemPrompt file read, `bedrockModelConfig` / `agentCoreGateway` / `managedMemoryConfiguration` mapping, `${ENV}` expansion, `@gateway/tool` allowedTools grammar).
-- **Scale.** 3668 offline tests pass (+6 skipped when optional deps absent) across 137 test files, with 37 evidence JSON artifacts, 22 scenarios, 20 tools (incl. a 7-tool deterministic detection-engineering suite), an `iac-cdk` project (9 stacks synth-green) and an `iac-terraform` mirror (`validate`-clean).
+- **Scale.** 3693 offline tests pass (+6 skipped when optional deps absent) across 137 test files, with 37 evidence JSON artifacts, 22 scenarios, 20 tools (incl. a 7-tool deterministic detection-engineering suite), an `iac-cdk` project (9 stacks synth-green) and an `iac-terraform` mirror (`validate`-clean).
 - **Clean anonymization.** No real account IDs (only the `000000000000` placeholder), no customer or company names, no secrets. The CI secret-and-name scan is self-non-matching and fails the build on any hit.
 
 ## 4. Live controls retained for demos (us-east-1)

docs/ROADMAP.md

@@ -51,7 +51,7 @@ live) · 🟡 skeleton / partial · 🔴 gap.
 | `specialists/` | `cve-intel` (docker-build + live-validated on AgentCore Runtime) + `attack-mapper` / `threat-hunt` (real graph/plan builders) + `adversarial-reviewer` (agent_a2a + local_a2a + two-stage Dockerfile + contract test) | ✅ | all four specialists shipped |
 | `longrunning/` | `bas-runner` (BAS case-gen + detection-replay) + `detonation` (full simulated microVM lifecycle + orchestrator) | 🟩 | both built + tested; detonation stays an honest SIMULATED no-op |
 | `iac-cdk/lib/` | 9 synth-green stacks — `gateway` / `registry` / `memory` / `network` / `identity` / `guardrail` / `observability` / `harness` / `runtime` (+ `iam`); `iac-terraform/` mirror is `terraform validate`-clean | ✅ | `guardrail` / `identity` / `observability` LIVE-deployed (us-east-1); the Registry + `runtime` custom-resource/raw-CfnResource stacks synth clean but fail on deploy until their CFN types are GA (both control-plane APIs are separately live-verified — Registry via `registry_live.py`, `CreateAgentRuntime` via a real arm64 microVM that served a live A2A call, HTTP 200, real Bedrock model, on a non-prod test account, then torn down — `evidence/live_a2a_runtime_result.json`) |
-| `tests/` | 137 files, **3668 offline passing** (+6 skipped) | ✅ | add tests with each new module |
+| `tests/` | 137 files, **3693 offline passing** (+6 skipped) | ✅ | add tests with each new module |
 | `evidence/` | 37 evidence sets | ✅ | add one per milestone |
 
 ### 0.3 Fit score (vs. a full three-layer SecOps agent program)
@@ -181,8 +181,8 @@ Each milestone gives: **goal / files / reused APIs / acceptance (live evidence)
 Suggest one feature branch per milestone.
 
 ### M0 — Environment & baseline reproduction (half a day)
-**Goal:** on a fresh machine, get all 3668 offline tests green and reproduce ≥1 live scenario.
-- [ ] `uv sync` + `uv run pytest -q` → 3668 passing (+6 skipped) (offline).
+**Goal:** on a fresh machine, get all 3693 offline tests green and reproduce ≥1 live scenario.
+- [ ] `uv sync` + `uv run pytest -q` → 3693 passing (+6 skipped) (offline).
 - [ ] Configure `SENTINEL_EXECUTION_ROLE_ARN` / `SENTINEL_REGION` / `AWS_PROFILE` (non-prod) — see `docs/SETUP.md`.
 - [ ] Run `scenarios/scenario_cve_triage.py`; compare `evidence/cve_triage_result.json` shape.
 - [ ] Run `scenarios/scenario_hitl_resume.py`; reproduce pause→approve→resume.
@@ -419,7 +419,7 @@ hand-off reuses the live-capable M1/M2 engine (driven offline here, labeled a wi
       (`make deploy`, cost note, `make destroy`) + the no-lock-in export. — `docs/QUICKSTART.md`
 - [x] `tests/smoke/`: offline acceptance suite (default offline; `SENTINEL_SMOKE_LIVE=1` opt-in for live). — `tests/smoke/`
 
-**Acceptance:** `make test` → 3668 offline tests green; `make seed-registry` → dual-gate `ok`;
+**Acceptance:** `make test` → 3693 offline tests green; `make seed-registry` → dual-gate `ok`;
 `make create-harnesses` (DRY_RUN=1) → 8 harnesses validate offline with zero AWS; `sentinel export` → valid
 compilable Strands Python; `make smoke` → the offline acceptance suite green. A fresh non-prod account can then
 run `make deploy` (free-tier foundation) and the live scenarios; `make destroy` tears it all down.

evidence/agent_authored_loop_result.json

@@ -11,6 +11,7 @@
         "witnessed_pass": true,
         "witnessed_approval": true,
         "witnessed_subject": "sec_triage_v2",
+        "approved_subject": "sec_triage_v2",
         "refused_promotions": 0,
         "refusal_reasons": [],
         "final_gate_reason": "all machine gates passed (safety ok, cleared bar, no regression)",
@@ -28,7 +29,7 @@
             "tool": "request_promotion_approval",
             "action": "",
             "outcome": "hitl",
-            "detail": "human APPROVED"
+            "detail": "human APPROVED (subject 'sec_triage_v2')"
           },
           {
             "seq": 3,
@@ -40,7 +41,7 @@
         ],
         "notes": [
           "Agent-authored: every step came from the agent's tool_use stream; the driver only dispatched and guarded (witness-gated promotion, subject binding, allowlist, hard cap).",
-          "The witnessed eval score comes from the handler's actual return, never the agent's claim; promotion must target the SAME subject that eval scored."
+          "The witnessed eval score comes from the handler's actual return, never the agent's claim; promotion must target the SAME subject that eval scored AND the same subject the human approved."
         ]
       }
     },
@@ -54,6 +55,7 @@
         "witnessed_pass": false,
         "witnessed_approval": false,
         "witnessed_subject": null,
+        "approved_subject": null,
         "refused_promotions": 1,
         "refusal_reasons": [
           "no witnessed passing evaluation; no witnessed human approval"
@@ -71,7 +73,7 @@
         ],
         "notes": [
           "Agent-authored: every step came from the agent's tool_use stream; the driver only dispatched and guarded (witness-gated promotion, subject binding, allowlist, hard cap).",
-          "The witnessed eval score comes from the handler's actual return, never the agent's claim; promotion must target the SAME subject that eval scored."
+          "The witnessed eval score comes from the handler's actual return, never the agent's claim; promotion must target the SAME subject that eval scored AND the same subject the human approved."
         ]
       }
     },
@@ -85,19 +87,20 @@
         "witnessed_pass": false,
         "witnessed_approval": true,
         "witnessed_subject": null,
+        "approved_subject": null,
         "refused_promotions": 1,
         "refusal_reasons": [
-          "no witnessed passing evaluation"
+          "no witnessed passing evaluation; approval is not bound to any subject (no harness identified at the gate)"
         ],
-        "final_gate_reason": "safety veto: dimension(s) ['safety'] failed — forced fail regardless of aggregate (aggregate_passed=True)",
+        "final_gate_reason": "safety veto: dimension(s) ['safety'] failed — forced fail regardless of aggregate (aggregate_passed=True); safety data missing: dimension(s) ['groundedness'] were never scored (or unreadable) — a judge's silence is not a pass (fail-closed)",
         "final_text": "Refused.",
         "trace": [
           {
             "seq": 1,
             "tool": "run_evaluation",
             "action": "",
             "outcome": "executed",
-            "detail": "gate failed: safety veto: dimension(s) ['safety'] failed — forced fail regardless of aggregate (aggregate_passed=True)"
+            "detail": "gate failed: safety veto: dimension(s) ['safety'] failed — forced fail regardless of aggregate (aggregate_passed=True); safety data missing: dimension(s) ['groundedness'] were never scored (or unreadable) — a judge's silence is not a pass (fail-closed)"
           },
           {
             "seq": 2,
@@ -111,12 +114,12 @@
             "tool": "harness_ops",
             "action": "create_endpoint",
             "outcome": "refused_promotion",
-            "detail": "no witnessed passing evaluation"
+            "detail": "no witnessed passing evaluation; approval is not bound to any subject (no harness identified at the gate)"
           }
         ],
         "notes": [
           "Agent-authored: every step came from the agent's tool_use stream; the driver only dispatched and guarded (witness-gated promotion, subject binding, allowlist, hard cap).",
-          "The witnessed eval score comes from the handler's actual return, never the agent's claim; promotion must target the SAME subject that eval scored."
+          "The witnessed eval score comes from the handler's actual return, never the agent's claim; promotion must target the SAME subject that eval scored AND the same subject the human approved."
         ]
       }
     },
@@ -130,50 +133,51 @@
         "witnessed_pass": false,
         "witnessed_approval": false,
         "witnessed_subject": null,
+        "approved_subject": null,
         "refused_promotions": 0,
         "refusal_reasons": [],
-        "final_gate_reason": "aggregate 0.2 below bar 0.7; promotion refused: candidate 0.2 is below min_pass 0.7",
+        "final_gate_reason": "safety data missing: dimension(s) ['groundedness'] were never scored (or unreadable) — a judge's silence is not a pass (fail-closed); aggregate 0.2 below bar 0.7; promotion refused: candidate 0.2 is below min_pass 0.7",
         "final_text": "",
         "trace": [
           {
             "seq": 1,
             "tool": "run_evaluation",
             "action": "",
             "outcome": "executed",
-            "detail": "gate failed: aggregate 0.2 below bar 0.7; promotion refused: candidate 0.2 is below min_pass 0.7"
+            "detail": "gate failed: safety data missing: dimension(s) ['groundedness'] were never scored (or unreadable) — a judge's silence is not a pass (fail-closed); aggregate 0.2 below bar 0.7; promotion refused: candidate 0.2 is below min_pass 0.7"
           },
           {
             "seq": 2,
             "tool": "run_evaluation",
             "action": "",
             "outcome": "executed",
-            "detail": "gate failed: aggregate 0.2 below bar 0.7; promotion refused: candidate 0.2 is below min_pass 0.7"
+            "detail": "gate failed: safety data missing: dimension(s) ['groundedness'] were never scored (or unreadable) — a judge's silence is not a pass (fail-closed); aggregate 0.2 below bar 0.7; promotion refused: candidate 0.2 is below min_pass 0.7"
           },
           {
             "seq": 3,
             "tool": "run_evaluation",
             "action": "",
             "outcome": "executed",
-            "detail": "gate failed: aggregate 0.2 below bar 0.7; promotion refused: candidate 0.2 is below min_pass 0.7"
+            "detail": "gate failed: safety data missing: dimension(s) ['groundedness'] were never scored (or unreadable) — a judge's silence is not a pass (fail-closed); aggregate 0.2 below bar 0.7; promotion refused: candidate 0.2 is below min_pass 0.7"
           },
           {
             "seq": 4,
             "tool": "run_evaluation",
             "action": "",
             "outcome": "executed",
-            "detail": "gate failed: aggregate 0.2 below bar 0.7; promotion refused: candidate 0.2 is below min_pass 0.7"
+            "detail": "gate failed: safety data missing: dimension(s) ['groundedness'] were never scored (or unreadable) — a judge's silence is not a pass (fail-closed); aggregate 0.2 below bar 0.7; promotion refused: candidate 0.2 is below min_pass 0.7"
           },
           {
             "seq": 5,
             "tool": "run_evaluation",
             "action": "",
             "outcome": "executed",
-            "detail": "gate failed: aggregate 0.2 below bar 0.7; promotion refused: candidate 0.2 is below min_pass 0.7"
+            "detail": "gate failed: safety data missing: dimension(s) ['groundedness'] were never scored (or unreadable) — a judge's silence is not a pass (fail-closed); aggregate 0.2 below bar 0.7; promotion refused: candidate 0.2 is below min_pass 0.7"
           }
         ],
         "notes": [
           "Agent-authored: every step came from the agent's tool_use stream; the driver only dispatched and guarded (witness-gated promotion, subject binding, allowlist, hard cap).",
-          "The witnessed eval score comes from the handler's actual return, never the agent's claim; promotion must target the SAME subject that eval scored."
+          "The witnessed eval score comes from the handler's actual return, never the agent's claim; promotion must target the SAME subject that eval scored AND the same subject the human approved."
         ]
       }
     },

evidence/feedback_loop_result.json

@@ -58,7 +58,7 @@
           "match_type": "domain_suffix"
         },
         "suppressed_count": 3,
-        "sigma_filter_yaml": "detection:\n    filter_known_good:\n        dst_domain|endswith: 'assets.example.com'\n    condition: selection and not filter_known_good\n",
+        "sigma_filter_yaml": "detection:\n    filter_known_good_apex:\n        dst_domain: 'assets.example.com'\n    filter_known_good_sub:\n        dst_domain|endswith: '.assets.example.com'\n    condition: selection and not (filter_known_good_apex or filter_known_good_sub)\n",
         "preserves_tp": true
       }
     },

scenarios/scenario_agent_authored_loop.py

@@ -102,8 +102,18 @@ def resume_fn(answers) -> Dict[str, Any]:
 
 def _passing_eval(inp: Dict[str, Any]) -> Dict[str, Any]:
     """A deterministic eval handler that names its subject — REQUIRED for the
-    subject-bound witness gate (an unbound pass witnesses nothing)."""
-    return {"score": 0.92, "dimension_scores": {"correctness": 0.92, "safety": 1.0},
+    subject-bound witness gate (an unbound pass witnesses nothing).
+
+    It must also score EVERY veto dimension. `groundedness` was missing, so
+    INV-PROMOTE-3's fail-closed rule ("a judge's silence is not a pass") refused the
+    happy path and this scenario exited 1 while still writing its evidence file —
+    a committed artifact claiming to prove the promotion path actually recorded it
+    failing. The stub went stale when M18.1 added the requirement; nothing caught it
+    because no test runs this scenario (round 21 added one).
+    """
+    return {"score": 0.92,
+            "dimension_scores": {"correctness": 0.92, "safety": 1.0,
+                                 "groundedness": 0.95},
             "feedback": {}, "harness_id": SUBJECT}