Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
<img alt="python" src="https://img.shields.io/badge/python-3.10%2B-2997ff"/>
<img alt="bedrock-agentcore" src="https://img.shields.io/badge/Amazon%20Bedrock-AgentCore%20Harness-ff9900"/>
<img alt="version" src="https://img.shields.io/badge/version-0.4.0-2997ff"/>
<img alt="tests" src="https://img.shields.io/badge/offline%20tests-3715%20passing-1D8102"/>
<img alt="tests" src="https://img.shields.io/badge/offline%20tests-3725%20passing-1D8102"/>
<img alt="coverage" src="https://img.shields.io/badge/coverage-90%25-1D8102"/>
<img alt="milestones" src="https://img.shields.io/badge/milestones-M0--M15%20delivered-1D8102"/>
<img alt="hardening" src="https://img.shields.io/badge/adversarial%20audit-100%20defects%20fixed-8b5cf6"/>
Expand Down Expand Up @@ -99,7 +99,7 @@ Honest build status per capability — mirrors the self-audit.
| **Tools** | `nvd_lookup` / `epss_kev` / `attack_lookup` / `web_search` | 🟡 **reference stubs** (offline-safe, contract-tested) | `tools/`, `tests/test_tool_handlers.py` |
| **Tools** | `siem_query` / `asset_lookup` / `enrich_ioc` / `ops_query` — backend-pluggable | 🟢 **built + tested** (offline mock default; `*_LIVE`=1 switches to a real stdlib-HTTP client — env-driven URL + bearer, timeouts, all failures→`upstream_error` with no silent fallback — proven end-to-end against an in-process 127.0.0.1 mock server, zero external network) | `tools/{siem_query,asset_lookup,enrich_ioc,ops_query}/`, `tests/test_*_live.py` |

🟢 built & validated · 🟡 built, partial · 🟠 designed with loadable config · ⚪ design narrative only. **3715 offline tests pass** (+6 skipped when optional deps absent).
🟢 built & validated · 🟡 built, partial · 🟠 designed with loadable config · ⚪ design narrative only. **3725 offline tests pass** (+6 skipped when optional deps absent).

## 🚀 Quickstart

Expand Down Expand Up @@ -247,7 +247,7 @@ Borrowed patterns (see [`docs/BLUEPRINT.md`](docs/BLUEPRINT.md)): supervisor→s
| [`docs/GOVERNANCE.md`](docs/GOVERNANCE.md) | Registry dual-gate, HITL, sandbox hooks, and tag-guard controls |
| [`docs/COMPLIANCE.md`](docs/COMPLIANCE.md) | Capability → SOC 2 / ISO 27001 / NIST CSF 2.0 control mapping (anchors machine-verified) |
| [`docs/OBSERVABILITY.md`](docs/OBSERVABILITY.md) | Logging (`logutil`), metrics (token/latency/tool-call/error/eval), the OTEL/Transaction-Search path |
| [`docs/TESTING.md`](docs/TESTING.md) | The 3715-test offline suite: layout, determinism, how to run |
| [`docs/TESTING.md`](docs/TESTING.md) | The 3725-test offline suite: layout, determinism, how to run |
| [`docs/FIDELITY-REPORT.md`](docs/FIDELITY-REPORT.md) | The self-audit — real vs. built vs. designed, with limits stated |
| [`docs/ROADMAP.md`](docs/ROADMAP.md) | Delivered milestones (M0–M12) and what's next |
| [**API reference (live)**](https://aws-samples.github.io/sample-sentinel-harness/) | Rendered `sentinel_harness` API docs (pdoc → GitHub Pages) |
Expand Down Expand Up @@ -285,7 +285,7 @@ sentinel-harness/
├── iac-cdk/ L3 CDK stacks (9; guardrail/identity/obs/vpc live) 🟢
├── iac-terraform/ deployable Terraform mirror (validate-clean) 🟢
├── docs/ QUICKSTART · ARCHITECTURE · BLUEPRINT · SETUP · HARNESSES · GOVERNANCE · TESTING · FIDELITY-REPORT · ROADMAP
├── tests/ offline unit + config tests (3715) 🟢
├── tests/ offline unit + config tests (3725) 🟢
└── .github/workflows/ CI incl. a customer-name / secret gate
```

Expand Down
2 changes: 1 addition & 1 deletion docs/FIDELITY-REPORT.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ This is real, not narration:
- **Layer 3 foundation ships runnable, tested code.** The dual-gate tool/skill registry, PreToolUse sandbox hook, and Agent Factory (fleet provision, dry-run, cross-env tag-guard) are built and unit-tested.
- **Native IaC, not hand-rolled resources.** The gateway / registry / memory / harness stacks use the native `AWS::BedrockAgentCore::*` CloudFormation types. Per the README status matrix, the **Gateway and Memory CFN types are registered**; the Registry type is not yet in CFN (see limitations).
- **Config path works.** `pip install -e .` succeeds (`[tool.setuptools] packages = ["sentinel_harness", "intake"]`), the `sentinel` console script works, and `sentinel create <harness.yaml>` loads real config via `sentinel_harness/loader.py` (systemPrompt file read, `bedrockModelConfig` / `agentCoreGateway` / `managedMemoryConfiguration` mapping, `${ENV}` expansion, `@gateway/tool` allowedTools grammar).
- **Scale.** 3715 offline tests pass (+6 skipped when optional deps absent) across 137 test files, with 37 evidence JSON artifacts, 22 scenarios, 20 tools (incl. a 7-tool deterministic detection-engineering suite), an `iac-cdk` project (9 stacks synth-green) and an `iac-terraform` mirror (`validate`-clean).
- **Scale.** 3725 offline tests pass (+6 skipped when optional deps absent) across 137 test files, with 37 evidence JSON artifacts, 22 scenarios, 20 tools (incl. a 7-tool deterministic detection-engineering suite), an `iac-cdk` project (9 stacks synth-green) and an `iac-terraform` mirror (`validate`-clean).
- **Clean anonymization.** No real account IDs (only the `000000000000` placeholder), no customer or company names, no secrets. The CI secret-and-name scan is self-non-matching and fails the build on any hit.

## 4. Live controls retained for demos (us-east-1)
Expand Down
8 changes: 4 additions & 4 deletions docs/ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ live) · 🟡 skeleton / partial · 🔴 gap.
| `specialists/` | `cve-intel` (docker-build + live-validated on AgentCore Runtime) + `attack-mapper` / `threat-hunt` (real graph/plan builders) + `adversarial-reviewer` (agent_a2a + local_a2a + two-stage Dockerfile + contract test) | ✅ | all four specialists shipped |
| `longrunning/` | `bas-runner` (BAS case-gen + detection-replay) + `detonation` (full simulated microVM lifecycle + orchestrator) | 🟩 | both built + tested; detonation stays an honest SIMULATED no-op |
| `iac-cdk/lib/` | 9 synth-green stacks — `gateway` / `registry` / `memory` / `network` / `identity` / `guardrail` / `observability` / `harness` / `runtime` (+ `iam`); `iac-terraform/` mirror is `terraform validate`-clean | ✅ | `guardrail` / `identity` / `observability` LIVE-deployed (us-east-1); the Registry + `runtime` custom-resource/raw-CfnResource stacks synth clean but fail on deploy until their CFN types are GA (both control-plane APIs are separately live-verified — Registry via `registry_live.py`, `CreateAgentRuntime` via a real arm64 microVM that served a live A2A call, HTTP 200, real Bedrock model, on a non-prod test account, then torn down — `evidence/live_a2a_runtime_result.json`) |
| `tests/` | 137 files, **3715 offline passing** (+6 skipped) | ✅ | add tests with each new module |
| `tests/` | 137 files, **3725 offline passing** (+6 skipped) | ✅ | add tests with each new module |
| `evidence/` | 37 evidence sets | ✅ | add one per milestone |

### 0.3 Fit score (vs. a full three-layer SecOps agent program)
Expand Down Expand Up @@ -181,8 +181,8 @@ Each milestone gives: **goal / files / reused APIs / acceptance (live evidence)
Suggest one feature branch per milestone.

### M0 — Environment & baseline reproduction (half a day)
**Goal:** on a fresh machine, get all 3715 offline tests green and reproduce ≥1 live scenario.
- [ ] `uv sync` + `uv run pytest -q` → 3715 passing (+6 skipped) (offline).
**Goal:** on a fresh machine, get all 3725 offline tests green and reproduce ≥1 live scenario.
- [ ] `uv sync` + `uv run pytest -q` → 3725 passing (+6 skipped) (offline).
- [ ] Configure `SENTINEL_EXECUTION_ROLE_ARN` / `SENTINEL_REGION` / `AWS_PROFILE` (non-prod) — see `docs/SETUP.md`.
- [ ] Run `scenarios/scenario_cve_triage.py`; compare `evidence/cve_triage_result.json` shape.
- [ ] Run `scenarios/scenario_hitl_resume.py`; reproduce pause→approve→resume.
Expand Down Expand Up @@ -419,7 +419,7 @@ hand-off reuses the live-capable M1/M2 engine (driven offline here, labeled a wi
(`make deploy`, cost note, `make destroy`) + the no-lock-in export. — `docs/QUICKSTART.md`
- [x] `tests/smoke/`: offline acceptance suite (default offline; `SENTINEL_SMOKE_LIVE=1` opt-in for live). — `tests/smoke/`

**Acceptance:** `make test` → 3715 offline tests green; `make seed-registry` → dual-gate `ok`;
**Acceptance:** `make test` → 3725 offline tests green; `make seed-registry` → dual-gate `ok`;
`make create-harnesses` (DRY_RUN=1) → 8 harnesses validate offline with zero AWS; `sentinel export` → valid
compilable Strands Python; `make smoke` → the offline acceptance suite green. A fresh non-prod account can then
run `make deploy` (free-tier foundation) and the live scenarios; `make destroy` tears it all down.
Expand Down
77 changes: 73 additions & 4 deletions sentinel_harness/core.py
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@
import os, json, time, uuid
import boto3
from botocore.config import Config
from typing import Any

from .logutil import get_logger

Expand All @@ -47,8 +48,71 @@
# with _data (bounded connect/read + retries) instead of relying on boto defaults.
_CONTROL_CONFIG = Config(read_timeout=60, connect_timeout=15, retries={"max_attempts": 3})

_control = boto3.client("bedrock-agentcore-control", region_name=REGION, config=_CONTROL_CONFIG)
_data = boto3.client("bedrock-agentcore", region_name=REGION, config=_DATA_CONFIG)
class _LazyClient:
"""A boto3 client built on FIRST USE, not at import time.

Why this exists. `boto3.client("bedrock-agentcore-control")` costs ~2.1s and
`boto3.client("bedrock-agentcore")` ~2.0s — service models with 153 operations get
parsed and the credential chain resolved. Constructing both at import time meant a
bare ``import sentinel_harness`` took **4.5s at 7% CPU**: not computing, waiting.

Three consequences, the third being the one that matters most:

1. Every subprocess test paid it. 26 of them (the scenario + README-CLI E2E layers)
cost ~4.5s each — about 115s of a 202s suite, more than half, spent on work
nothing asked for.
2. **Importing a library had side effects.** `import sentinel_harness` read
``~/.aws/``, and on an EC2 host the credential chain can reach the instance
metadata service. A library must not touch the network or the user's credential
store just to be imported.
3. It forced the testing style. Because the real clients already existed by the time
any test ran, the only way to substitute a fake was to patch the module global
after the fact — which ~48 test files do.

Deliberately a proxy rather than a function, so nothing about the call sites or the
tests changes: `_control.create_harness(...)` still works, and
`monkeypatch.setattr(core, "_control", fake)` still replaces the whole object. Sibling
modules that bind `from .core import _control` get this proxy, and `set_region` still
rebinds them explicitly.

Thread-safety is deliberately NOT added. Two threads racing the first access would
build two clients and one would be discarded — wasteful, never wrong, since boto3
clients are independent. A lock here would be a hot-path cost paid by every call to
protect against a harmless race.

No ``__slots__``, deliberately. The first version had it and broke
``test_gateway.py::test_scenario_named_supervisor_imports_without_aws``, which patches
a method ON the client object (``setattr(sh._control, "create_gateway", ...)``) rather
than replacing the client wholesale. That is a legitimate existing pattern — it proves
importing a scenario makes no AWS call — and supporting it is part of the contract this
proxy has to keep. Three instances exist per process, so the memory `__slots__` would
save is irrelevant next to breaking a caller.
"""

def __init__(self, service: str, config: Config) -> None:
self._service = service
self._config = config
self._client: Any = None

def _resolve(self) -> Any:
if self._client is None:
self._client = boto3.client(
self._service, region_name=REGION, config=self._config
)
return self._client

def __getattr__(self, name: str) -> Any:
# __slots__ names are handled by the descriptors, so anything reaching here is a
# boto3 API surface (`create_harness`, `exceptions`, `meta`, ...).
return getattr(self._resolve(), name)

def __repr__(self) -> str:
state = "built" if self._client is not None else "not yet built"
return f"<_LazyClient {self._service!r} ({state})>"


_control: Any = _LazyClient("bedrock-agentcore-control", _CONTROL_CONFIG)
_data: Any = _LazyClient("bedrock-agentcore", _DATA_CONFIG)


def set_region(region: str) -> None:
Expand All @@ -66,8 +130,13 @@ def set_region(region: str) -> None:
raise ValueError("region must be a non-empty string")
REGION = region
os.environ["SENTINEL_REGION"] = region
_control = boto3.client("bedrock-agentcore-control", region_name=region, config=_CONTROL_CONFIG)
_data = boto3.client("bedrock-agentcore", region_name=region, config=_DATA_CONFIG)
# Lazy, like the import-time pair: `set_region` is called by the CLI's `--region`
# flag before any AWS work happens, so building the clients here would put the ~4.2s
# construction back on the startup path of every command — including the offline
# detection commands, which never touch AWS at all. `_LazyClient` reads the
# module-global REGION when it resolves, which this function has already updated.
_control = _LazyClient("bedrock-agentcore-control", _CONTROL_CONFIG)
_data = _LazyClient("bedrock-agentcore", _DATA_CONFIG)
# Sibling modules bind `from .core import _control` at import time, so their
# local name still points at the OLD client after we reassign ours. Rebind the
# borrowers explicitly (best-effort: they may not be imported yet) so a runtime
Expand Down
Loading