Skip to content

fix: the scanner and sweep reported infected repos as clean, in seven different ways - #3

Merged
wezell merged 7 commits into
mainfrom
fix-sweep-silent-clean
Aug 31, 2026
Merged

wezell merged 7 commits into
mainfrom
fix-sweep-silent-clean

Conversation

@fabrizzio-dotCMS

@fabrizzio-dotCMS fabrizzio-dotCMS commented Aug 27, 2026 •

Copy link
Copy Markdown
Member

Everything in this repo fails toward "clean". Seven defects were found in a single day — four in the tooling, three introduced while fixing it — and every one reported an infected file or repository as CLEAN. None ever produced a false alarm. That asymmetry is what this PR is written against.

Folded #5 and #6 in here so this is one review, not four.

1. Four ways a poisoned repo was reported clean

  • The sweep called rg with no fallback. Where rg is only a shell function — common; command -v rg succeeds in an interactive zsh and fails inside the script's bash — the command failed, $hits came back empty, and every repo printed "HEAD clean".
  • Section 1c was silently dead on every host without ripgrep: the grep branch spelled the alternation as ${REGEX_PATS[0]}|…|${REGEX_PATS[3]} against a 3-element array, so under set -u the subshell died before grep ran and the section printed nothing.
  • Two false positives were masking a real detection miss.
  • The campaign tag is a build number and it has already rotated — 9-3727-2, A9-3727-2, A9-3727-3 across a four-file sample set. A literals-only rule would have missed the third. The marker is now the generalised A?#-####-# pattern, pinned by a test against a build we have never seen.

2. There is no CLEAN verdict any more

The sweep clones with --mirror, which fetches refs. An object reachable from no ref is never looked at, and GitHub keeps serving those objects by SHA long after a squash orphans them. Four payload blobs were retrieved that way from repos this sweep had called CLEAN — ovh-k8s-cluster and support among them, the latter confirmed to be the whole variant C implant.

The scan was not wrong. The word was.

RESULT <repo> INFECTED|NO_REF_HITS|UNKNOWN <hits>

Every non-abort exit path now prints what the run does not cover. --deep does not close the gap either — git rev-list --all still walks only what is reachable. Orphaned objects cannot be enumerated at all, so verifying them is a per-SHA check against the incident catalogue, not a scan.

The caveat is a function called from both the org path and --local. Written inline in the summary it left --local silently without it — the same failure in miniature, a partial result read as a total one. The test caught that, not review.

3. One source of truth for the rules

The rules lived in polinrider_scan.sh, org_sweep.sh, and a third hardcoded copy inside classify_config(). They had drifted: the sweep matched the tag as the literal A9-3727 long after the scanner generalised it, leaving it blind to variant A on every run it ever performed.

Not a flat list — the two tools have different false-positive surfaces and that difference was deliberate:

set contents scope
core rmcej%otb%, wuqktam… hard everywhere
repo-only 'Sec-V', helloipbot, AUTH_API_KEY, 9-3727-2 hard in a repo sweep
host-only atob(process.env.AUTH_API_KEY hard on a workstation

Flattening them would promote 'Sec-V' to a hard indicator on developer machines, where it appears as quoted text in Claude Code transcripts — a recorded FP class that would produce false COMPROMISED verdicts.

4. Neither script will run unless its rules still match malware

The sweep has had an engine self-test since the rg failure. The host scanner never did, and it is the half that runs on developer machines. Both now plant known-positive carrier bytes, run their real rules with their real flags, and abort rather than report anything clean if a rule does not fire. With negative controls: a clean file must not match, and the node_modules exclude must still apply — a rule that fires on everything detects nothing, and that failure is invisible without one.

canaries.sh is a separate file, and that is the whole mechanism. This went through two versions that were both false, and neither failed loudly:

  1. Fixtures generated from ${SIGS[@]} — proves the rules match themselves; passes with every signature silently rewritten.
  2. Real carrier bytes hardcoded in rules.sh — one find-and-replace rewrote a signature and the sample proving it worked in the same pass.

Evidence that moves whenever the claim moves is not evidence. Both defects were found by the sabotage test, which breaks one rule at a time in a throwaway copy and requires the scanner to refuse to run.

Also fixed: a sourced file that returns does not abort its caller, so a missing canaries.sh let the scan proceed with the canary silently evaporated. Both scripts check on their own side and exit 2.

5. CI

The repo had none. Its suites ran when someone remembered — in a repo whose own history includes a suite that had no assertions and passed for weeks while testing nothing. Suites are discovered, not listed, so a new one cannot be silently skipped. Both engines run, because the grep/ripgrep difference has hidden a real detection failure before.

Validation

Every rule here is checked against four byte-exact carriers supplied by the IR lead on 2026-08-27 (SHA256-verified, held outside any repo). The claims in the comments are measurements, not inferences from a write-up — including two corrections: the padding run is 507 bytes, not the ~5,000 the old comment claimed, and the payload starts at column 510 in two carriers and 536 in the third, so the rule anchors on the run length and never on a column.

Five suites, all passing.

Known gap, recorded rather than solved

rules.sh is the single source for bash consumers. The PR-check service cannot source a bash file. That needs a canonical format both can read, decided once we know what the service is written in. Do not close it by copying the strings — that recreates exactly the drift this PR removes.

fabrizzio-dotCMS and others added 3 commits August 27, 2026 13:13
org_sweep.sh had three independent ways to return "clean" on an infected
repository. Each is fixed here with a regression test, because the previous
round of this work shipped a test suite that had no assertions and always
exited 0.

Silent-clean bugs fixed:

- The sweep called `rg` with no fallback and discarded its stderr. Where rg is
  only a shell function -- `command -v rg` succeeds in zsh and fails inside
  bash, which is the case on at least one engineer workstation -- the command
  failed, $hits came back empty, and every repo printed "HEAD clean (hard
  IOCs)". Reproduced against a known-infected fixture: the sweep called it
  clean. Replaced with git grep, which is also what makes ref-wide scanning
  cheap, plus a self-test that aborts rather than report anything as clean if
  a rule cannot match its own canary.

- The campaign tag was grepped as the literal 'A9-3727'. That is the
  obfuscator.io build; the older _$_ shuffle build writes
  global['!']='9-3727-2' with no A prefix and was invisible to the sweep.
  Now matched as a pattern (global.i / global['!'] assignment of A?#-####-#),
  which also survives build-number rotation. Verified: the old pattern misses
  the no-prefix build, the new one catches both.

- Whitespace padding was checked with `tail -c 2000`, but the padding sits at
  the START of the payload, so the last 2000 bytes are pure payload with no
  spaces at all. Now checked over the whole file at a 200-space threshold
  (observed at 507). This is not the rejected line-length heuristic:
  minification strips whitespace, so vendored bundles have long lines and no
  long space runs -- pinned by a negative-control test.

Coverage gaps closed:

- Enumerates every non-archived repo in both orgs via the API instead of 11
  hardcoded names. Self-propagation targets any repo the victim can push to.
- Clones --mirror and scans all refs, so refs/pull/* is covered. A normal
  clone does not fetch them, and poisoned blobs are still sitting there.
- Adds the 'Sec-V' quoted key and bare AUTH_API_KEY to the signature set. The
  bare form is what sees the .env carrier; atob(process.env.* cannot.
- Refs are scanned in batches so large repos do not blow the argument list.

Reporting:

- UNKNOWN is now distinct from CLEAN. A repo that failed to clone or whose
  engine errored has been proven nothing. Exit 1 infected, 2 not-scanned.
- Ref counts, batch counts, clone failures and suppressions are all printed.
  A silent cap reads as coverage.
- Verdicts are tallied through a file, not a shell variable: scan_repo runs in
  a subshell, so the previous counter was lost and the summary would have
  reported zero infected regardless of findings.
- Token now comes from the environment or `gh auth token`, not argv, where it
  was visible to every user on the box via ps.

Adds an allowlist that is scoped to paths, never to repositories, and that
still prints what it suppresses -- the detection tooling matches its own
indicators, so without it a nightly sweep over 355 repos is permanently red.

test_org_sweep.sh: 12 assertions, all passing, including a negative control
that fails against the previous implementation.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Same failure direction as the sweep fixes in the previous commit: the section
produced no output and the scan continued to a clean verdict.

The grep branch of section 1c spelled its alternation out by index,

    "${REGEX_PATS[0]}|${REGEX_PATS[1]}|${REGEX_PATS[2]}|${REGEX_PATS[3]}"

against a three-element array. Under `set -u` that is an unbound variable, so
the subshell died before grep ran and the section reported nothing. It only
ever took the rg branch on hosts with a ripgrep BINARY -- and rg is frequently
just a shell function, so `command -v rg` succeeds in an interactive zsh and
fails inside this script, which is exactly the case on at least one engineer
workstation. Reproduced against a file carrying two of the three patterns:
section 1c reported no loader-family patterns.

The alternation is now built from the array with IFS, so it cannot drift out of
sync with its length again, and stderr is no longer discarded in that branch --
an engine failure prints instead of reading as "no findings".

Also fixes the third pattern. It was `Sec-V:`, but the bytes in the payload are
the quoted object key `'Sec-V':`, so the quote is part of the indicator: the
unquoted form matches zero real payloads. Verified both ways against a fixture.

test_rg_detection.sh gains three assertions for section 1c, including a negative
control pinning the quoting. They fail against the previous implementation and
pass against this one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Validated against the four byte-exact carriers on 2026-08-27. The set contains
THREE distinct build IDs, not one:

    '9-3727-2'    tailwind.config.js   wave-1 shuffle
    "A9-3727-2"   karma.conf.js        obfuscator.io
    "A9-3727-3"   eslint.config.mjs    obfuscator.io

So the campaign tag is a build number that has already rotated, not a fixed
marker -- which is also why 'Sec-V' reads as the sole detector for the eslint
carrier. The marker is not absent there; it changed.

That matters because the wave-1 carrier is found by that number ALONE. It
carries no 'Sec-V' and no AUTH_API_KEY. Had it shipped as 9-3727-3, a
literals-only rule would have missed it completely.

The generalised marker already in this branch matches all three, including the
rotated one. This test pins that: it asserts first that no literal matches its
fixture, so the case cannot quietly stop proving anything, then that the marker
still fires.

15 assertions, all passing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
fabrizzio-dotCMS and others added 4 commits August 28, 2026 11:37
The sweep clones with --mirror, which fetches refs. An object reachable from
no ref is never looked at. GitHub keeps serving those objects by SHA long
after a squash orphans them, and four payload blobs were retrieved that way
from repos this sweep had reported CLEAN -- ovh-k8s-cluster and support among
them, the latter confirmed to be the whole variant C implant.

The scan was not wrong. The word was. CLEAN reads as "the implant is gone";
what the run actually proves is "no ref in this repo reaches the implant".
Those are different claims and the gap between them is where two poisoned
repos got written off.

So there is no CLEAN verdict any more:

  RESULT <repo> INFECTED|NO_REF_HITS|UNKNOWN <hits>

and every non-abort exit path now prints what the run does not cover --
orphaned objects, which --deep does not reach either, since `git rev-list
--all` still walks only what is reachable. They cannot be enumerated at all,
so verifying them is a per-SHA check against the incident catalogue, not a
scan. Saying that in the output is the point: no future reader should have to
rediscover it.

The caveat is a function called from both the org path and --local. Writing it
inline in the summary left --local silently without it, which is the same
failure in miniature -- a partial result read as a total one. The test caught
that, not review.

test_org_sweep.sh pins all three: the NO_REF_HITS token, the absence of any
CLEAN verdict, and the caveat being printed. 18 assertions, all passing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The rules lived in polinrider_scan.sh and org_sweep.sh separately, plus a third
hardcoded copy inside classify_config(). They drifted: the sweep matched the
campaign tag as the literal 'A9-3727' long after the scanner had generalised it,
which left the sweep blind to the wave-1 variant on every run it ever performed.
Nobody noticed, because a rule that has drifted still reports CLEAN.

rules.sh is now the only place a rule is written down. Both scripts source it and
abort if it is missing.

It is not a flat list. The two tools have different false-positive surfaces and
that difference was deliberate, so the file carries three sets:

  core        rmcej%otb%, wuqktam...        hard everywhere
  repo-only   'Sec-V', helloipbot,          hard in a repo sweep
              AUTH_API_KEY, 9-3727-2
  host-only   atob(process.env.AUTH_API_KEY hard on a workstation

Flattening these would have promoted 'Sec-V' to a hard indicator on developer
machines, where it appears as quoted text in Claude Code transcripts -- a
recorded FP class that would produce false COMPROMISED verdicts.

Three things surfaced while doing this:

- Neither tool carried the bare 9-3727-2 literal. Both relied on the marker
  pattern alone, which requires the global-assignment context to match. The
  handoff lists that literal as load-bearing. It is now in the repo set.
- classify_config() held a third copy of the signatures, inline.
- The comment claiming the payload line carries ~5,000 padding spaces was wrong.
  It is 507, measured on all three obfuscated carriers. The conclusion that
  comment supported was right; the number was not.

test_rules_single_source.sh, 21 assertions: both sets non-empty, the shared
indicators in both, the deliberate split preserved in each direction, no script
defining a rule inline, both scripts sourcing and aborting without rules.sh, and
the marker matching every known build plus a rotated one it has never seen.

All five suites pass. Verified against the four byte-exact carriers: all four
detected, marker fires on all three obfuscated builds including A9-3727-3.

Known gap, recorded rather than solved: rules.sh is the single source for BASH
consumers. A non-bash consumer -- the PR-check service -- cannot source it. That
needs a canonical format both can read, decided when we know what the service is
written in. Do not solve it by copying the strings.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…l match malware

org_sweep.sh has had an engine self-test since the rg-without-fallback failure.
polinrider_scan.sh never did, and it is the half that runs on developer
machines, where the environment is least predictable and the same failure is
waiting: `rg` is frequently only a shell function, so `command -v rg` succeeds
in an interactive zsh and the binary is absent inside this bash. The command
fails, output comes back empty, and empty reads as "nothing found".

Both scripts now plant known-positive carrier bytes, run their real rules with
their real flags, and abort rather than report anything clean if a rule does not
fire. There are negative controls too: a clean file must not match, and the
node_modules exclude must still apply -- a rule that fires on everything detects
nothing, and that failure is invisible without one.

canaries.sh, and why it is a separate file

This went through two versions that were both false, and neither failed loudly:

  1. Fixtures generated from ${SIGS[@]}. That proves the rules match themselves.
     It passes with every signature silently rewritten.
  2. Real carrier bytes, hardcoded in rules.sh. One find-and-replace across that
     file rewrote a signature AND the sample proving it worked, in the same
     pass. The self-test kept passing on a rule that matched no real carrier.

Evidence that moves whenever the claim moves is not evidence. The samples now
live in canaries.sh, out of reach of an edit to rules.sh, and that separation is
the entire mechanism rather than a matter of tidiness. rules.sh sources it, so
consumers still have one thing to source.

Both defects were found by the sabotage test, not by reading the diff. That test
breaks one rule at a time in a throwaway copy and requires the scanner to refuse
to run; it is the only thing standing between this canary and a third false
version of it.

Also fixed while here: a sourced file that `return`s does NOT abort its caller,
so a missing canaries.sh let the scan proceed with the canary silently
evaporated. Both scripts now check on their own side and exit 2.

Five suites pass. The sweep's sigs.js fixture is kept as well -- built from the
rule set on purpose, since it proves the engine can match literals containing
quotes and $, which is a different claim from the rules being right.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This repository has no CI. `gh pr checks` reports zero checks on every open PR,
so the four fixture suites run only when someone remembers to run them.

That is a poor fit for what they guard. Every defect these suites pin fails in
the same direction -- reporting an infected file or repository as CLEAN -- so a
regression here does not break a build, it quietly stops finding malware. The
repo has already been bitten by exactly this: test_config_classification.sh had
no assertions at all and passed for weeks while testing nothing, which is
recorded in f5bc89e.

Suites are DISCOVERED (test_*.sh at the root), not listed. Listing them means a
new suite silently never runs until someone edits this file, and it also breaks
the other way: test_org_sweep.sh does not exist on main yet -- it arrives with
PR #3 -- so a workflow naming it would fail on main for the wrong reason. The
job errors out if it discovers zero suites, so "passed by finding nothing"
cannot happen either.

Runs on a matrix of both search engines. The scanner falls back to grep where no
ripgrep BINARY exists, because `rg` is frequently only a shell function and
`command -v rg` can succeed in an interactive zsh while failing inside the
script. That exact difference hid a real detection failure, so the grep path
needs to be covered rather than assumed.

Also runs bash -n over every script. Syntax only: these are bash-specific and
macOS bash-3.2 compatible on purpose, so a style linter would fight the code.

actions/checkout is pinned by SHA, verified to resolve to v5.0.0.

Verified locally: the three suites on main are discovered and pass.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@fabrizzio-dotCMS fabrizzio-dotCMS changed the title fix: four ways the sweep and scanner reported infected repos as clean fix: the scanner and sweep reported infected repos as clean, in seven different ways Aug 28, 2026

@wezell wezell left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice fixes

@wezell
wezell merged commit 823e2fa into main Aug 31, 2026
3 checks passed
@fabrizzio-dotCMS
fabrizzio-dotCMS deleted the fix-sweep-silent-clean branch September 1, 2026 20:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants