diff --git a/README.md b/README.md index 1f1b218..3b69307 100644 --- a/README.md +++ b/README.md @@ -13,7 +13,7 @@ python bedrock-agentcore version - tests + tests coverage milestones hardening @@ -99,7 +99,7 @@ Honest build status per capability β€” mirrors the self-audit. | **Tools** | `nvd_lookup` / `epss_kev` / `attack_lookup` / `web_search` | 🟑 **reference stubs** (offline-safe, contract-tested) | `tools/`, `tests/test_tool_handlers.py` | | **Tools** | `siem_query` / `asset_lookup` / `enrich_ioc` / `ops_query` β€” backend-pluggable | 🟒 **built + tested** (offline mock default; `*_LIVE`=1 switches to a real stdlib-HTTP client β€” env-driven URL + bearer, timeouts, all failuresβ†’`upstream_error` with no silent fallback β€” proven end-to-end against an in-process 127.0.0.1 mock server, zero external network) | `tools/{siem_query,asset_lookup,enrich_ioc,ops_query}/`, `tests/test_*_live.py` | -🟒 built & validated Β· 🟑 built, partial Β· 🟠 designed with loadable config Β· βšͺ design narrative only. **3715 offline tests pass** (+6 skipped when optional deps absent). +🟒 built & validated Β· 🟑 built, partial Β· 🟠 designed with loadable config Β· βšͺ design narrative only. **3725 offline tests pass** (+6 skipped when optional deps absent). ## πŸš€ Quickstart @@ -247,7 +247,7 @@ Borrowed patterns (see [`docs/BLUEPRINT.md`](docs/BLUEPRINT.md)): supervisorβ†’s | [`docs/GOVERNANCE.md`](docs/GOVERNANCE.md) | Registry dual-gate, HITL, sandbox hooks, and tag-guard controls | | [`docs/COMPLIANCE.md`](docs/COMPLIANCE.md) | Capability β†’ SOC 2 / ISO 27001 / NIST CSF 2.0 control mapping (anchors machine-verified) | | [`docs/OBSERVABILITY.md`](docs/OBSERVABILITY.md) | Logging (`logutil`), metrics (token/latency/tool-call/error/eval), the OTEL/Transaction-Search path | -| [`docs/TESTING.md`](docs/TESTING.md) | The 3715-test offline suite: layout, determinism, how to run | +| [`docs/TESTING.md`](docs/TESTING.md) | The 3725-test offline suite: layout, determinism, how to run | | [`docs/FIDELITY-REPORT.md`](docs/FIDELITY-REPORT.md) | The self-audit β€” real vs. built vs. designed, with limits stated | | [`docs/ROADMAP.md`](docs/ROADMAP.md) | Delivered milestones (M0–M12) and what's next | | [**API reference (live)**](https://aws-samples.github.io/sample-sentinel-harness/) | Rendered `sentinel_harness` API docs (pdoc β†’ GitHub Pages) | @@ -285,7 +285,7 @@ sentinel-harness/ β”œβ”€β”€ iac-cdk/ L3 CDK stacks (9; guardrail/identity/obs/vpc live) 🟒 β”œβ”€β”€ iac-terraform/ deployable Terraform mirror (validate-clean) 🟒 β”œβ”€β”€ docs/ QUICKSTART Β· ARCHITECTURE Β· BLUEPRINT Β· SETUP Β· HARNESSES Β· GOVERNANCE Β· TESTING Β· FIDELITY-REPORT Β· ROADMAP -β”œβ”€β”€ tests/ offline unit + config tests (3715) 🟒 +β”œβ”€β”€ tests/ offline unit + config tests (3725) 🟒 └── .github/workflows/ CI incl. a customer-name / secret gate ``` diff --git a/docs/FIDELITY-REPORT.md b/docs/FIDELITY-REPORT.md index f56ea08..1abc8b4 100644 --- a/docs/FIDELITY-REPORT.md +++ b/docs/FIDELITY-REPORT.md @@ -29,7 +29,7 @@ This is real, not narration: - **Layer 3 foundation ships runnable, tested code.** The dual-gate tool/skill registry, PreToolUse sandbox hook, and Agent Factory (fleet provision, dry-run, cross-env tag-guard) are built and unit-tested. - **Native IaC, not hand-rolled resources.** The gateway / registry / memory / harness stacks use the native `AWS::BedrockAgentCore::*` CloudFormation types. Per the README status matrix, the **Gateway and Memory CFN types are registered**; the Registry type is not yet in CFN (see limitations). - **Config path works.** `pip install -e .` succeeds (`[tool.setuptools] packages = ["sentinel_harness", "intake"]`), the `sentinel` console script works, and `sentinel create ` loads real config via `sentinel_harness/loader.py` (systemPrompt file read, `bedrockModelConfig` / `agentCoreGateway` / `managedMemoryConfiguration` mapping, `${ENV}` expansion, `@gateway/tool` allowedTools grammar). -- **Scale.** 3715 offline tests pass (+6 skipped when optional deps absent) across 137 test files, with 37 evidence JSON artifacts, 22 scenarios, 20 tools (incl. a 7-tool deterministic detection-engineering suite), an `iac-cdk` project (9 stacks synth-green) and an `iac-terraform` mirror (`validate`-clean). +- **Scale.** 3725 offline tests pass (+6 skipped when optional deps absent) across 137 test files, with 37 evidence JSON artifacts, 22 scenarios, 20 tools (incl. a 7-tool deterministic detection-engineering suite), an `iac-cdk` project (9 stacks synth-green) and an `iac-terraform` mirror (`validate`-clean). - **Clean anonymization.** No real account IDs (only the `000000000000` placeholder), no customer or company names, no secrets. The CI secret-and-name scan is self-non-matching and fails the build on any hit. ## 4. Live controls retained for demos (us-east-1) diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 85676b1..628ce32 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -51,7 +51,7 @@ live) Β· 🟑 skeleton / partial Β· πŸ”΄ gap. | `specialists/` | `cve-intel` (docker-build + live-validated on AgentCore Runtime) + `attack-mapper` / `threat-hunt` (real graph/plan builders) + `adversarial-reviewer` (agent_a2a + local_a2a + two-stage Dockerfile + contract test) | βœ… | all four specialists shipped | | `longrunning/` | `bas-runner` (BAS case-gen + detection-replay) + `detonation` (full simulated microVM lifecycle + orchestrator) | 🟩 | both built + tested; detonation stays an honest SIMULATED no-op | | `iac-cdk/lib/` | 9 synth-green stacks β€” `gateway` / `registry` / `memory` / `network` / `identity` / `guardrail` / `observability` / `harness` / `runtime` (+ `iam`); `iac-terraform/` mirror is `terraform validate`-clean | βœ… | `guardrail` / `identity` / `observability` LIVE-deployed (us-east-1); the Registry + `runtime` custom-resource/raw-CfnResource stacks synth clean but fail on deploy until their CFN types are GA (both control-plane APIs are separately live-verified β€” Registry via `registry_live.py`, `CreateAgentRuntime` via a real arm64 microVM that served a live A2A call, HTTP 200, real Bedrock model, on a non-prod test account, then torn down β€” `evidence/live_a2a_runtime_result.json`) | -| `tests/` | 137 files, **3715 offline passing** (+6 skipped) | βœ… | add tests with each new module | +| `tests/` | 137 files, **3725 offline passing** (+6 skipped) | βœ… | add tests with each new module | | `evidence/` | 37 evidence sets | βœ… | add one per milestone | ### 0.3 Fit score (vs. a full three-layer SecOps agent program) @@ -181,8 +181,8 @@ Each milestone gives: **goal / files / reused APIs / acceptance (live evidence) Suggest one feature branch per milestone. ### M0 β€” Environment & baseline reproduction (half a day) -**Goal:** on a fresh machine, get all 3715 offline tests green and reproduce β‰₯1 live scenario. -- [ ] `uv sync` + `uv run pytest -q` β†’ 3715 passing (+6 skipped) (offline). +**Goal:** on a fresh machine, get all 3725 offline tests green and reproduce β‰₯1 live scenario. +- [ ] `uv sync` + `uv run pytest -q` β†’ 3725 passing (+6 skipped) (offline). - [ ] Configure `SENTINEL_EXECUTION_ROLE_ARN` / `SENTINEL_REGION` / `AWS_PROFILE` (non-prod) β€” see `docs/SETUP.md`. - [ ] Run `scenarios/scenario_cve_triage.py`; compare `evidence/cve_triage_result.json` shape. - [ ] Run `scenarios/scenario_hitl_resume.py`; reproduce pauseβ†’approveβ†’resume. @@ -419,7 +419,7 @@ hand-off reuses the live-capable M1/M2 engine (driven offline here, labeled a wi (`make deploy`, cost note, `make destroy`) + the no-lock-in export. β€” `docs/QUICKSTART.md` - [x] `tests/smoke/`: offline acceptance suite (default offline; `SENTINEL_SMOKE_LIVE=1` opt-in for live). β€” `tests/smoke/` -**Acceptance:** `make test` β†’ 3715 offline tests green; `make seed-registry` β†’ dual-gate `ok`; +**Acceptance:** `make test` β†’ 3725 offline tests green; `make seed-registry` β†’ dual-gate `ok`; `make create-harnesses` (DRY_RUN=1) β†’ 8 harnesses validate offline with zero AWS; `sentinel export` β†’ valid compilable Strands Python; `make smoke` β†’ the offline acceptance suite green. A fresh non-prod account can then run `make deploy` (free-tier foundation) and the live scenarios; `make destroy` tears it all down. diff --git a/sentinel_harness/core.py b/sentinel_harness/core.py index ba85f7b..196f547 100644 --- a/sentinel_harness/core.py +++ b/sentinel_harness/core.py @@ -34,6 +34,7 @@ import os, json, time, uuid import boto3 from botocore.config import Config +from typing import Any from .logutil import get_logger @@ -47,8 +48,71 @@ # with _data (bounded connect/read + retries) instead of relying on boto defaults. _CONTROL_CONFIG = Config(read_timeout=60, connect_timeout=15, retries={"max_attempts": 3}) -_control = boto3.client("bedrock-agentcore-control", region_name=REGION, config=_CONTROL_CONFIG) -_data = boto3.client("bedrock-agentcore", region_name=REGION, config=_DATA_CONFIG) +class _LazyClient: + """A boto3 client built on FIRST USE, not at import time. + + Why this exists. `boto3.client("bedrock-agentcore-control")` costs ~2.1s and + `boto3.client("bedrock-agentcore")` ~2.0s β€” service models with 153 operations get + parsed and the credential chain resolved. Constructing both at import time meant a + bare ``import sentinel_harness`` took **4.5s at 7% CPU**: not computing, waiting. + + Three consequences, the third being the one that matters most: + + 1. Every subprocess test paid it. 26 of them (the scenario + README-CLI E2E layers) + cost ~4.5s each β€” about 115s of a 202s suite, more than half, spent on work + nothing asked for. + 2. **Importing a library had side effects.** `import sentinel_harness` read + ``~/.aws/``, and on an EC2 host the credential chain can reach the instance + metadata service. A library must not touch the network or the user's credential + store just to be imported. + 3. It forced the testing style. Because the real clients already existed by the time + any test ran, the only way to substitute a fake was to patch the module global + after the fact β€” which ~48 test files do. + + Deliberately a proxy rather than a function, so nothing about the call sites or the + tests changes: `_control.create_harness(...)` still works, and + `monkeypatch.setattr(core, "_control", fake)` still replaces the whole object. Sibling + modules that bind `from .core import _control` get this proxy, and `set_region` still + rebinds them explicitly. + + Thread-safety is deliberately NOT added. Two threads racing the first access would + build two clients and one would be discarded β€” wasteful, never wrong, since boto3 + clients are independent. A lock here would be a hot-path cost paid by every call to + protect against a harmless race. + + No ``__slots__``, deliberately. The first version had it and broke + ``test_gateway.py::test_scenario_named_supervisor_imports_without_aws``, which patches + a method ON the client object (``setattr(sh._control, "create_gateway", ...)``) rather + than replacing the client wholesale. That is a legitimate existing pattern β€” it proves + importing a scenario makes no AWS call β€” and supporting it is part of the contract this + proxy has to keep. Three instances exist per process, so the memory `__slots__` would + save is irrelevant next to breaking a caller. + """ + + def __init__(self, service: str, config: Config) -> None: + self._service = service + self._config = config + self._client: Any = None + + def _resolve(self) -> Any: + if self._client is None: + self._client = boto3.client( + self._service, region_name=REGION, config=self._config + ) + return self._client + + def __getattr__(self, name: str) -> Any: + # __slots__ names are handled by the descriptors, so anything reaching here is a + # boto3 API surface (`create_harness`, `exceptions`, `meta`, ...). + return getattr(self._resolve(), name) + + def __repr__(self) -> str: + state = "built" if self._client is not None else "not yet built" + return f"<_LazyClient {self._service!r} ({state})>" + + +_control: Any = _LazyClient("bedrock-agentcore-control", _CONTROL_CONFIG) +_data: Any = _LazyClient("bedrock-agentcore", _DATA_CONFIG) def set_region(region: str) -> None: @@ -66,8 +130,13 @@ def set_region(region: str) -> None: raise ValueError("region must be a non-empty string") REGION = region os.environ["SENTINEL_REGION"] = region - _control = boto3.client("bedrock-agentcore-control", region_name=region, config=_CONTROL_CONFIG) - _data = boto3.client("bedrock-agentcore", region_name=region, config=_DATA_CONFIG) + # Lazy, like the import-time pair: `set_region` is called by the CLI's `--region` + # flag before any AWS work happens, so building the clients here would put the ~4.2s + # construction back on the startup path of every command β€” including the offline + # detection commands, which never touch AWS at all. `_LazyClient` reads the + # module-global REGION when it resolves, which this function has already updated. + _control = _LazyClient("bedrock-agentcore-control", _CONTROL_CONFIG) + _data = _LazyClient("bedrock-agentcore", _DATA_CONFIG) # Sibling modules bind `from .core import _control` at import time, so their # local name still points at the OLD client after we reassign ours. Rebind the # borrowers explicitly (best-effort: they may not be imported yet) so a runtime diff --git a/tests/test_lazy_clients.py b/tests/test_lazy_clients.py new file mode 100644 index 0000000..f08a52e --- /dev/null +++ b/tests/test_lazy_clients.py @@ -0,0 +1,291 @@ +""" +Importing the library must be cheap and side-effect free. +======================================================== +A fifth test sweep read the suite's own timing data β€” `--durations` β€” and found every +subprocess test paying a flat ~4.5s that was not work: + + import sentinel_harness 4.50s at 7% CPU (not computing, waiting) + python -c "pass" 0.03s + +`-X importtime` put 4.15s of self time in `sentinel_harness.core`, and the cause was two +module-level statements: + + _control = boto3.client("bedrock-agentcore-control", ...) # 2.15s + _data = boto3.client("bedrock-agentcore", ...) # 2.01s + +The cost is NOT service-model parsing, which was my first assumption and is wrong β€” those +models are gzipped JSON on disk and load in milliseconds. Measured in matched environments: + + no credentials 4.46s the full chain, ending at IMDS + fake credentials 0.31s first provider hits + AWS_EC2_METADATA_DISABLED=true 0.32s + +It is the **instance-metadata probe timing out**. So importing this library made it reach +for ``169.254.169.254`` β€” the exact address its own INV-EGRESS family exists to refuse. +Three consequences, the third mattering most: + +1. **Cost.** 26 subprocess tests x 4.5s ~ 115s of a 202s suite β€” more than half the + runtime spent building clients nothing used. +2. **Importing had SIDE EFFECTS.** `import sentinel_harness` read ``~/.aws/``, and on an + EC2 host the credential chain can reach the instance metadata service. A library must + not touch the network or a user's credential store merely to be imported. That is a + correctness and least-surprise property, not a performance one. +3. **It shaped the tests.** Because real clients existed before any test ran, the only way + to substitute a fake was to patch the module global afterwards β€” which ~48 files do. + +`core._LazyClient` defers construction to first attribute access. Import dropped +4.50s -> 0.26s (17x) and the suite 202s -> 105s (1.9x), with no call-site or test changes. + +This file pins both properties. The performance bound is deliberately loose (a CI runner is +slower and noisier than a laptop) β€” it is set to catch a REGRESSION to eager construction, +which costs seconds, not to police tenths. +""" +from __future__ import annotations + +import subprocess +import textwrap + +import pytest + +import child_pytest + +# Eager construction cost ~4.5s; lazy is ~0.3s. A 2.0s ceiling cannot be tripped by +# ordinary noise and cannot be satisfied by the eager version even on a fast machine. +_IMPORT_BUDGET_SECONDS = 2.0 + + +def _run_snippet(code: str, timeout: float = 120) -> subprocess.CompletedProcess: + """Run `code` in a clean child interpreter through the shared launcher. + + Uses `child_pytest.resolve_python_launcher()` rather than a hardcoded `uv`/`python`: + that mistake has been made five times in this repo (see tests/child_pytest.py). + """ + launcher = child_pytest.resolve_python_launcher() + return subprocess.run( + [*launcher, "-c", textwrap.dedent(code)], + cwd=child_pytest.REPO_ROOT, capture_output=True, text=True, timeout=timeout, + ) + + +# --------------------------------------------------------------------------- # +# Importing is cheap # +# --------------------------------------------------------------------------- # +def test_importing_the_package_is_fast(): + """Measured in a FRESH interpreter: an in-process timer would see boto3's warm caches + and report a fast import even if construction were eager again.""" + result = _run_snippet( + """ + import time + t = time.perf_counter() + import sentinel_harness # noqa: F401 + print(f"{time.perf_counter() - t:.3f}") + """ + ) + assert result.returncode == 0, (result.stdout + result.stderr)[-600:] + elapsed = float(result.stdout.strip().splitlines()[-1]) + assert elapsed < _IMPORT_BUDGET_SECONDS, ( + f"importing sentinel_harness took {elapsed:.2f}s, over the " + f"{_IMPORT_BUDGET_SECONDS}s budget. Something is doing real work at import time β€” " + "most likely a boto3 client built at module level again. Use core._LazyClient." + ) + + +def test_importing_core_does_not_build_a_client(): + """The structural version of the same property, immune to timing noise.""" + result = _run_snippet( + """ + from sentinel_harness import core + print(repr(core._control)) + print(repr(core._data)) + """ + ) + assert result.returncode == 0, (result.stdout + result.stderr)[-600:] + for line in result.stdout.strip().splitlines()[-2:]: + assert "not yet built" in line, ( + f"a client was constructed at import time: {line}. Import must stay free of " + "AWS work β€” the credential chain can reach the instance metadata service." + ) + + +def test_importing_makes_no_aws_call(): + """Property 2, enforced rather than described: fail the import if boto3 is asked to + build ANY client while `sentinel_harness` is being imported.""" + result = _run_snippet( + """ + import boto3 + real = boto3.client + + def _forbidden(*a, **k): + raise AssertionError(f"import built a boto3 client: {a[:1]}") + + boto3.client = _forbidden + import sentinel_harness # must not construct anything + from sentinel_harness import gateway, registry_live, mcp_server # noqa: F401 + boto3.client = real + print("OK") + """ + ) + assert result.returncode == 0, ( + "importing the package (or gateway / registry_live / mcp_server) constructed a " + f"boto3 client:\n{(result.stdout + result.stderr)[-700:]}" + ) + assert "OK" in result.stdout + + +# --------------------------------------------------------------------------- # +# The proxy is transparent β€” every contract its callers already relied on # +# --------------------------------------------------------------------------- # +class TestTheProxyIsTransparent: + """~48 test files patch these objects. The proxy has to keep all of it working.""" + + def test_replacing_the_whole_client_still_works(self, monkeypatch): + """The dominant pattern: `monkeypatch.setattr(core, "_control", fake)`.""" + from sentinel_harness import core + + class _Fake: + def list_harnesses(self, **_kw): + return {"harnesses": ["fake"]} + + monkeypatch.setattr(core, "_control", _Fake()) + assert core._control.list_harnesses()["harnesses"] == ["fake"] + + def test_patching_a_method_on_the_client_still_works(self): + """`test_gateway.py::test_scenario_named_supervisor_imports_without_aws` patches a + method ON the client. The first version of the proxy used `__slots__` and broke it β€” + recorded because it is exactly the kind of caller a "transparent" proxy must not + surprise.""" + from sentinel_harness.core import _LazyClient, _CONTROL_CONFIG + + proxy = _LazyClient("bedrock-agentcore-control", _CONTROL_CONFIG) + proxy.create_gateway = lambda **_kw: "patched" + assert proxy.create_gateway() == "patched" + assert "not yet built" in repr(proxy), ( + "patching a method triggered construction β€” the patch should shadow the " + "forwarded attribute without resolving the client" + ) + + def test_internal_fields_are_not_forwarded(self): + """`_service` / `_config` / `_client` must resolve on the proxy itself. Forwarding + them would recurse or silently build a client to answer a bookkeeping question.""" + from sentinel_harness.core import _LazyClient, _DATA_CONFIG + + proxy = _LazyClient("bedrock-agentcore", _DATA_CONFIG) + assert proxy._service == "bedrock-agentcore" + assert proxy._client is None + assert "not yet built" in repr(proxy) + + def test_set_region_stays_lazy(self, monkeypatch): + """`set_region` is called by the CLI's --region flag before any AWS work β€” including + for the offline detection commands, which never touch AWS. Building clients there + would put the ~4.2s cost back on every command's startup path.""" + from sentinel_harness import core + + monkeypatch.setattr(core, "_control", core._LazyClient("x", core._CONTROL_CONFIG)) + monkeypatch.setattr(core, "_data", core._LazyClient("y", core._DATA_CONFIG)) + monkeypatch.setenv("SENTINEL_REGION", "us-east-1") + core.set_region("eu-west-1") + try: + assert core.REGION == "eu-west-1" + assert "not yet built" in repr(core._control) + assert "not yet built" in repr(core._data) + finally: + core.set_region("us-east-1") + + def test_the_deferred_client_uses_the_current_region(self, monkeypatch): + """Laziness must not mean staleness: a client built AFTER `set_region` has to use + the new region, not the one that was current at import.""" + from sentinel_harness import core + + built = {} + + def _fake_boto_client(service, region_name=None, config=None): + built["service"] = service + built["region"] = region_name + return object() + + monkeypatch.setattr(core.boto3, "client", _fake_boto_client) + monkeypatch.setattr(core, "REGION", "ap-southeast-2") + proxy = core._LazyClient("bedrock-agentcore-control", core._CONTROL_CONFIG) + proxy._resolve() + assert built["region"] == "ap-southeast-2", ( + f"the deferred client was built for {built['region']!r}; it must read the " + "module-global REGION at resolve time" + ) + + def test_the_client_is_built_once_and_cached(self, monkeypatch): + from sentinel_harness import core + + calls = [] + monkeypatch.setattr( + core.boto3, "client", + lambda service, **_kw: calls.append(service) or object(), + ) + proxy = core._LazyClient("bedrock-agentcore", core._DATA_CONFIG) + proxy._resolve() + proxy._resolve() + proxy._resolve() + assert len(calls) == 1, f"the client was rebuilt {len(calls)} times: {calls}" + + +# --------------------------------------------------------------------------- # +# The guard's own control # +# --------------------------------------------------------------------------- # +def test_construction_reaches_for_instance_metadata(): + """What the 4.1s actually WAS β€” and why this is a correctness finding, not a tuning one. + + My first version of this control asserted "eager construction costs > 2s" and FAILED, + reporting 0.09s. The premise was wrong, not the measurement: inside pytest the + credential chain is already satisfied (fake keys in the environment), so it short- + circuits. Measured properly, in matched environments: + + no credentials 4.46s the full chain, ending at IMDS + fake credentials 0.31s first provider hits + AWS_EC2_METADATA_DISABLED=true 0.32s + + So the cost is the **instance-metadata probe timing out** β€” `boto3.client()` at import + time made this library reach for ``169.254.169.254``, the exact address its own + INV-EGRESS family exists to refuse. Deferring construction means an import no longer + touches the credential store or the network at all. + + Asserts the RATIO between matched runs rather than an absolute, so it stays meaningful + on a fast CI runner and on a laptop. + """ + def _timed(env_extra: dict) -> float: + code = """ + import time, os + import boto3 + t = time.perf_counter() + boto3.client("bedrock-agentcore-control", region_name="us-east-1") + boto3.client("bedrock-agentcore", region_name="us-east-1") + print(f"{time.perf_counter() - t:.3f}") + """ + import os as _os + env = {k: v for k, v in _os.environ.items() + if not k.startswith(("AWS_", "SENTINEL_"))} + env["AWS_DEFAULT_REGION"] = "us-east-1" + env.update(env_extra) + launcher = child_pytest.resolve_python_launcher() + r = subprocess.run( + [*launcher, "-c", textwrap.dedent(code)], + cwd=child_pytest.REPO_ROOT, capture_output=True, text=True, + timeout=180, env=env, + ) + if r.returncode != 0: + pytest.skip(f"could not construct clients here: {r.stderr[-200:]}") + return float(r.stdout.strip().splitlines()[-1]) + + without_imds = _timed({"AWS_EC2_METADATA_DISABLED": "true", + "AWS_ACCESS_KEY_ID": "t", "AWS_SECRET_ACCESS_KEY": "t"}) + with_chain = _timed({}) # no credentials at all -> the chain walks to IMDS + + if with_chain < 1.0: + pytest.skip( + f"the credential chain resolved in {with_chain:.2f}s β€” this host answers or " + "refuses IMDS instantly, so the cost this guard describes is not reproducible " + "here. The laziness itself is still asserted by the tests above." + ) + assert with_chain > without_imds * 3, ( + f"constructing clients took {with_chain:.2f}s with the full credential chain vs " + f"{without_imds:.2f}s with IMDS disabled. The gap is the metadata probe, and it is " + "why client construction must not happen at import time." + )