test: guard the process-globals rounds 20-21 introduced - #30
Merged
Conversation
A fourth test sweep took three layers no previous sweep had: the installed-wheel surface,
mutation testing of five more invariant families, and process-global isolation. The first
two found nothing to fix; the third found a defect in the test I was writing.
**Installed-wheel E2E (8/8).** Built a wheel, installed it outside the repo, and exercised
what a pip user actually gets: the package + 5 submodules import, the registry resolves
from the INSTALLED copy (INV-MCP-2), the MCP gate obtains a real governance decision
(INV-MCP-1), `coerce_bool` rejects string truthiness (INV-COERCE), the egress guard refuses
every metadata spelling (INV-EGRESS), the `sentinel` console script runs, 4 tool handlers
import, and promotion is refused with no witnessed gate (INV-OPS-7). Every security
invariant holds on the real pip path, not just in a checkout.
**Mutation testing, 6/6 caught** across five families never mutated before — INV-PROMOTE-3
(both the missing-safety-dims check and its enabling switch), INV-SANDBOX-3, INV-GOV-4
(ledger truncation AND the anchored tail-hash comparison), INV-PLAY-1. With round 23 that
is 12/12 over 9 families. Same self-checking harness: a control mutation runs first and
nothing is reported unless it fails.
**The isolation guards, and the bug in them.** Rounds 20-21 added three process-globals,
each a way for one test to weaken a check for everything after it. The new file asserts
none leaks. Positive-controlling it found that my behavioural assertion was too weak:
assert out["ok"] is False # PASSED with the witness deliberately leaked
With the witness leaked the gate correctly OPENS, the call reaches the control plane, and
the handler returns ok:False with `upstream_error` — so the loose assertion passed for the
wrong reason while the gate was disabled. The environment assertion caught the planted
leak; the behavioural one did not. It now asserts the error KIND. Two assertions that look
like they test the same thing can differ in strength, and the weaker one passing is how a
real leak would ship.
That positive control also surfaced a self-harm risk worth stating plainly: with the
witness leaked, the handler issued a genuine `CreateHarnessEndpoint`. It was rejected
(`UnrecognizedClientException`) only because the ambient credentials happened to be
invalid. On a machine with working credentials, a test in a suite that advertises ZERO AWS
would have created a production endpoint. The control plane is now stubbed with an object
whose every attribute raises, so a leak is caught by assertion rather than by a network
round trip. Verified: no botocore frame appears, the stub intercepts, and the leak is
still detected.
The file is `zz_`-prefixed so the default alphabetical order collects it last — these
assertions are about the state the rest of the suite leaves behind. Under this repo's
`pytest-randomly` default that becomes a sample rather than a guarantee, which the module
docstring says outright rather than pretending otherwise.
Tested: 3715 passed / 6 skipped in BOTH fixed and random order; installed-wheel E2E 8/8;
scenario E2E 25/25; README CLI E2E 13/13; IaC E2E tsc 0 / 8-of-8 stack tests / synth 9
stacks; ruff clean; both mypy gates clean; make ci green.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A fourth test sweep took three layers no previous sweep had. The first two found nothing to fix; the third found a defect in the test I was writing.
1. Installed-wheel E2E — 8/8
Built a wheel, installed it outside the repo, and exercised what a pip user actually gets:
coerce_boolrejects string truthiness (INV-COERCE)sentinelconsole script runsEvery security invariant holds on the real pip path, not just in a checkout.
2. Mutation testing — 6/6 caught, five new families
autonomy: ignore missing safety dims (INV-PROMOTE-3)autonomy: disable the safety-dims switch (INV-PROMOTE-3)sandbox: allow any package source (INV-SANDBOX-3)provenance: allow a truncated ledger (INV-GOV-4)provenance: skip the anchored tail-hash check (INV-GOV-4)simulation: rename the checkpoint digest (INV-PLAY-1)With round 23 that is 12/12 over 9 invariant families. Same self-checking harness — a control mutation runs first and nothing is reported unless it fails.
3. The isolation guards, and the bug in them
Rounds 20–21 added three process-globals, each a way for one test to weaken a check for everything after it. The new file asserts none leaks.
Positive-controlling it found my behavioural assertion was too weak:
With the witness leaked the gate correctly opens, the call reaches the control plane, and the handler returns
ok: Falsewithupstream_error— so the loose assertion passed for the wrong reason while the gate was disabled. The environment assertion caught the planted leak; the behavioural one did not. It now asserts the error kind.Two assertions that look like they test the same thing can differ in strength, and the weaker one passing is how a real leak would ship.
A self-harm risk, stated plainly
That same positive control revealed that with the witness leaked, the handler issued a genuine
CreateHarnessEndpoint. It was rejected (UnrecognizedClientException) only because the ambient credentials happened to be invalid.On a machine with working credentials, a test in a suite that advertises ZERO AWS would have created a production endpoint.
The control plane is now stubbed with an object whose every attribute raises, so a leak is caught by assertion rather than by a network round trip. Verified: no botocore frame appears, the stub intercepts, and the leak is still detected.
Ordering, said outright
The file is
zz_-prefixed so the default alphabetical order collects it last — these assertions are about the state the rest of the suite leaves behind. Under this repo'spytest-randomlydefault that becomes a sample rather than a guarantee, which the module docstring says rather than pretending otherwise.Testing
tscexit 0 · 8/8 stack tests ·cdk synth9 stacksruffclean · both mypy gates clean ·make cigreen