Real atomics, real detections, and four new assurance gates - #64
Merged
Conversation
The pipeline's trust guarantees were documented but not enforced, so the tool's output could be mistaken for real detection evidence. - Canary gating now runs in every `run` path (runTasks) against the same Linux/Docker executor+collector the campaign uses. A run whose positive control does not fire is marked INCONCLUSIVE and its coverage is not presented as valid. Previously the canary ran only in the `canary` subcommand and no run was ever gated; model.Inconclusive was never assigned. - Dashboard canary status is the real result, not a hardcoded healthy:true. - Dry/synthetic runs are marked non-evidentiary end to end (CampaignResult.Synthetic): loud HTML/Navigator banner, and never appended to history.json or published to docs/data/coverage.json. - Reject partial container config so a real atomic is never paired with a synthetic collector. - Evidence window pads only the end, not the start (false DETECTED from an earlier run's events). - Collector: widen date pre-filter +/-1 day (non-UTC boundary); remove dead BaseURL/User/Pass fields and the unused canary dryRun param. - verify-lab.sh: source the indexer credential from env/secrets. - Refactor the four run* commands onto one runTasks helper. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Fidelity: only genuine process-creation events (Sysmon 1 / Security 4688, or the image+commandLine signature) are tagged process telemetry. Enumeration/ network events (e.g. 4798 callerProcessName) no longer credit a process-creation detection (false DETECTED on real telemetry). EventID is now carried into the normalized event. - Unsupported modifiers (base64/cidr/windash/...) → INCONCLUSIVE, not a silent MISSED. - Wildcards compile to anchored, case-insensitive regexps: `?` supported, `*\svchost.exe` no longer matches `...svchost.exe.malware`, metacharacters escaped; modifiers (contains/startswith/endswith) anchor correctly. - `N of prefix_*` / `all of prefix_*` expand by glob AND the parser no longer drops the trailing `and not filter` (was losing filter suppression). - `Field: null` matches absent/empty fields; `not(...)` (no space) parses; `of them` excludes filter/falsepositive identifiers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…coded one Every technique executed the same hardcoded "id; whoami". atomic_ids was parsed from plan YAML, carried through emulation stages and the arbiter feed, and written into the proof chain -- but never resolved to a command. A ten-technique campaign ran one command ten times and labelled the results T1082, T1033, T1518..., so every verdict but T1059.004's was about a command nobody asked for. The README's claim that Execute "runs Atomic Red Team tests" was not true of the code. New internal/atomic resolves each atomic_id against the pinned ART tree. Resolution is strict -- override, then vendored ART, then error -- with no default command, because a fabricated input produces a fabricated verdict. Fixing it immediately exposed two more defects: - Five of ten techniques in plans/discovery.yml named WINDOWS atomics while the campaign targets a Linux victim (T1082-1, T1033-1, T1007-1, T1016-1, T1049-1 are all command_prompt tests). Invisible while everything ran id; whoami. Plans now name the correct Linux tests and a platform mismatch is a loud ERROR. - T1059.004-1 writes a script that pings 8.8.8.8, traffic leaving the isolated lab network against the project's own containment rule. Its host input is pinned to loopback via an override that keeps the upstream command. Also: ART was recorded, not pinned. lab-fetch.sh cloned HEAD then overwrote the pin file with whatever it got, so a re-clone could silently change every command the engine runs. scripts/fetch-atomics.sh checks out the recorded commit and verifies HEAD matches. Adds SKIPPED_PREREQ: an atomic whose own prerequisites are unmet never ran, and is no longer reported as a detection miss. A test fails if any two techniques in the shipped campaign resolve to the same command, making the original defect unshippable. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Nine of the ten Linux Sigma rules were the identical rule. Every one matched "Image endswith /id or /whoami", differing only in title, because they were written against an engine that executed id; whoami for every technique. All nine shared one byte-identical positive fixture, so the "10 rules, 20 fixtures, CI enforced" detection-as-code claim was testing one rule nine times against one event. Each technique now has a detection for what its own atomic actually does, with distinct fixtures whose NEGATIVES include a neighbouring technique's activity -- so a rule that is secretly a copy fails its own negative test. win_proc_create.yml carried a non-UUID identifier (win-proc-create-001), a Sigma spec violation the hand-rolled Go parser accepted silently. Found by adding sigma check. Adds a data.sysmon normalizer path so a Linux-Sysmon source needs no Go changes: EventID 1 is process creation, any other Sysmon event is low fidelity and can never satisfy a process_creation rule. Deliberately not routed through data.win.*, which is Wazuh's Windows eventchannel convention -- using it for Linux would make the telemetry misreport its own origin. A test fails if any two rules share detection logic. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Retires the technique->rule map hardcoded in cmd/purpleloop, which duplicated
mappings/attack_rule_map.seed.json -- a file no code loaded. The mapping is now
data the repository can lint, extended with expected_rules[] so a technique is
DETECTED when ANY expected rule matches; asserting a single rule understated
coverage whenever a second rule was the one that legitimately caught the
behaviour.
Adds attribution labels to every verdict (window_and_host_scoped, window_scoped,
window_overlap, unscoped). The hard part already existed -- RunResult.Window pads
only the end, with a documented rationale -- what was missing was saying how
well-scoped each verdict is. Overlap is a campaign-level property, so it is
computed once every window is known. This changes no verdict; it states the
quality of the evidence behind one.
Adds detect_latency_ms from data already collected.
Report corrections:
- ERROR was counted as "missed" in the HTML headline, and SKIPPED_PREREQ/ERROR
sat inside the dashboard's coverage denominator. A harness failure is not a
detection gap; coverage is measured only over techniques that exercised one.
- campaign and build were hardcoded ("discovery", "v1.3.0") regardless of what
ran; both now come from the run.
- untested_rules is populated with a reason per rule, enforced by a CI lint, so
a rule can no longer be added, never exercised, and still count as coverage.
- gap.why is filled from the verdict's note instead of always being empty.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Replaces the fixed time.Sleep(10s) per technique and its 10-minute query window with collector.Settler, which polls until an expected rule matches or ingestion settles. The old wait was wrong in both directions: too long when the SIEM had already indexed the events (ten techniques spent 100s asleep), too short when it had not, producing a NO_TELEMETRY that was really an impatience artifact. Every real run now writes reports/runs/<id>/events.jsonl, and `purpleloop replay <run>` re-runs the REAL evaluator over it with no Docker and no Wazuh. This is the regression test the project could not previously have: CI has fixtures, but a fixture is a claim about the world written by the same person as the rule, not a sample of it. A rule edit can now be checked against every past run -- would this have broken a detection that already worked? The capture writer refuses synthetic runs, whose telemetry is fabricated and would be indistinguishable from a real dataset on replay, and skips a run that collected nothing rather than leaving an empty file that looks like a capture. testdata/sample-run ships a dataset in the capture format so replay is exercisable from a clean clone. Its README states plainly that it is not lab telemetry, and it deliberately includes a MISSED -- a sample where everything passes would teach the wrong thing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Coverage without a false-positive rate is half a metric: a rule that matches everything scores perfect coverage and is worthless. The repository's negative fixtures prove a rule rejects an event crafted to be rejected, which is a claim about the author's intent. This measures every rule against a workload written independently of it. emulation/benign-baseline.yml is 26 read-only administrator commands across seven categories, including adjacent cases -- benign uses of the very binaries the detections watch. Those are what make the test worth running: a rule keyed on a tool name rather than on what the tool is being asked to do fires there. It found two real false positives on introduction, both now fixed: - T1087.001 fired on `getent hosts localhost`, a DNS lookup, not account enumeration. It now requires an account database. - T1135 fired on a bare `mount`, routine administration. It now requires a network filesystem. `purpleloop precision` exits non-zero on any false positive, so it gates CI. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
runTasks hardcoded a Linux target, rules directory and executor. SSHExecutor and the Windows rule existed only inside the standalone `canary` subcommand, which also hardcoded the IP 192.168.88.13 -- so the v1.0 cross-platform coverage claim could not be exercised through `run` at all. Targets are declared in lab/targets.yml, techniques are dispatched to the target matching the platform their rule-map entry names, and the canary runs once per platform in use: a healthy Linux canary says nothing about whether Windows telemetry is flowing. Credentials come from the environment, never from source. `purpleloop diff <a> <b> [--fail-on-regression]` names the technique that regressed -- "T1082 DETECTED -> MISSED" -- instead of only moving a percentage. That was impossible while history.json stored a coverage number and nothing else; it now carries per-technique verdicts. Verdicts meaning "never exercised" share a rank, so ERROR -> SKIPPED_PREREQ is a change, not a regression. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Archive logging did not survive a restart. The collector reads archives.json, which the manager only writes when <logall_json> is yes; the image ships it as no. It had been enabled by hand INSIDE the running container, which cannot work: the manager bind-mounts /wazuh-config-mount/etc/ossec.conf from the host and its entrypoint copies that over /var/ossec/etc/ossec.conf on every start, so the edit is wiped by the very restart needed to apply it. The stack then comes up healthy, the agent enrols, and every technique reports NO_TELEMETRY -- which reads as a detection problem and is not one. scripts/enable-archives.sh (make archives) patches the host-side source of the config mount and then verifies the setting is live inside the container. The victim entrypoint hid a total telemetry failure: it ran service auditd start || auditd || true so every failure was swallowed. It now probes the kernel audit subsystem, loads execve rules when usable, and otherwise says loudly that this victim has no process-creation telemetry and that runs will be INCONCLUSIVE as a result. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CI ran only ./internal/... , skipping every guard in ./cmd/... -- atomic resolution, attribution, replay and diff. It now runs both. Adds four gates: - gofmt, which go vet does not cover - sigma check (pySigma), which enforces spec rules the hand-rolled Go parser tolerates; it caught a non-UUID rule identifier. The two excluded validators are SigmaHQ house conventions for their own repository, not the specification. - replay of testdata/sample-run: the real matcher over real event shapes - purpleloop precision: the false-positive gate `make atomics` is part of the build, not an optional step: without the pinned ART tree the engine has no command to run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
README documented behaviour the code did not have. It now describes strict atomic resolution, the three verdicts that mean "this never tested a detection" and are excluded from coverage, the override mechanism, and the new precision, replay and diff commands. RESULTS.md held a single JSON blob claiming T1059.004 -- a Linux Bash technique -- was DETECTED, with evidence being net.exe on Windows spawned by the Wazuh agent itself. Three things wrong with it, all consequences of the hardcoded id; whoami: wrong platform, wrong rule credited, and the "attack" was the monitoring agent's own activity. Replaced with a real run, and the old blob kept with an explanation, because a project that claims evidence should show what its own bad evidence looked like. PITFALLS records two measured environmental limits so they are not re-attempted blind: auditd cannot work on a host whose kernel exposes no audit sysctls (reproduced under --privileged --network host), and Sysmon for Linux runs only under systemd. Running Sysmon in a privileged container was built and rejected: its eBPF is kernel-global and the event carries no container identity, so scoping it either leaks host processes through PID reuse or drops the atomics' own short-lived ones. Both manufacture false verdicts. The fix is a Linux VM, as the Windows victim already is. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
jayelbotvibe-web
changed the title
The secrets job failed on every pull request with "GITHUB_TOKEN is now required to scan pull requests" -- a breaking change in gitleaks-action v2, not a finding. A secrets check that is permanently red teaches everyone to ignore it, which is worse than not running one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
192.168.88.13 was asserted as fact in seven places -- the canary's fallback, lab/targets.yml's default_host, an ssh.go comment, four spots in STARTUP.md and one in startup.sh. It only ever existed on one machine's bridged network. On any other host, bridged, NAT (vmnet8) and host-only (vmnet1) give different subnets, so the documented address is not merely absent but points somewhere else: the canary would probe a host that is not the victim, and the setup guide told you to expect an address you will never see. `canary` now reads the Windows host from lab/targets.yml like `run` does, and there is deliberately no default. With WINDOWS_SSH_HOST unset it says so and exits, naming the variable, rather than quietly probing a wrong address. STARTUP.md now tells you to find the address with ipconfig and export it, which is the only instruction that is true on every host. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A two-round code review found that purple-loop's trust guarantees were documented but not enforced in the Go that produces reports — the tool's output could be mistaken for real detection evidence. This PR enforces them and fixes the correctness bugs behind false verdicts.
🔴 Integrity contracts (were doc-only)
runpath against the same Linux/Docker executor+collector the campaign uses. A run whose positive control doesn't fire isINCONCLUSIVEand its coverage is not presented/published. (Previously the canary ran only in thecanarysubcommand; no run was ever gated;model.Inconclusivewas never assigned.)healthy:true.CampaignResult.Synthetic): loud HTML/Navigator banner, never appended tohistory.jsonor published todocs/data/coverage.json.🟠 Correctness (false verdicts)
INCONCLUSIVE, not silentMISSED.?supported, no trailing/leading slop, metachars escaped);N of prefix_*expands by glob and no longer dropsand not filter;Field: null,not(...), and filter-excludingof them.🟢 Hygiene
admin:SecretPasswordfromscripts/verify-lab.sh(now env/secrets-sourced).Tests
Full suite green (
go test ./... -count=1); added regression tests for anchored globs, aggregate-glob + trailing filter,Field: null,not(...), unsupported-modifier → INCONCLUSIVE, synthetic-not-published, and real canary status. Two existing tests updated to reflect corrected (not weakened) behavior; noted inline.🤖 Generated with Claude Code