Skip to content

ci: run the fixture suites on every change - #5

Closed
fabrizzio-dotCMS wants to merge 1 commit into
mainfrom
ci/run-test-suites
Closed

fabrizzio-dotCMS wants to merge 1 commit into
mainfrom
ci/run-test-suites

Conversation

@fabrizzio-dotCMS

Copy link
Copy Markdown
Member

Why

This repo has no CI. gh pr checks reports zero checks on every open PR, so the fixture suites run only when someone remembers to.

That is a poor fit for what they guard. Every defect these suites pin fails in the same direction — reporting an infected file or repository as CLEAN. A regression here does not break a build; it quietly stops finding malware.

The repo has already been bitten by exactly this. test_config_classification.sh had no assertions at all and passed for weeks while testing nothing — recorded in f5bc89e:

test_config_classification.sh had no assertions and always exited 0; when the pad and longline rules changed, both of its fixtures silently stopped testing anything.

What it runs

job what
Fixture suites (grep) every test_*.sh, no ripgrep installed
Fixture suites (ripgrep) every test_*.sh, ripgrep installed
Shell syntax bash -n over every *.sh

Both engines are in the matrix on purpose. The scanner falls back to grep where no ripgrep binary exists, because rg is frequently only a shell function — command -v rg can succeed in an interactive zsh and fail inside the script. That exact difference hid a real detection failure (section 1c returning nothing while the scan reported clean), so the grep path has to be covered rather than assumed.

Suites are discovered, not listed

Two failure modes avoided:

The job errors if it discovers zero suites, so "passed by finding nothing" cannot happen either. That is the same class of bug as everything else in this repo: a check that quietly succeeds by doing nothing.

Notes

  • actions/checkout is pinned by SHA, verified to resolve to v5.0.0.
  • POLINRIDER_DNS_TIMEOUT=1 bounds the scanner's unified-log pass. It is a no-op on Linux but must not be able to stall a runner; it is a documented 120s default that made a fixture run look hung.
  • bash -n only. These scripts are bash-specific and macOS bash-3.2 compatible by design, so a style linter would fight the code rather than help it.

Verified locally: the three suites currently on main are discovered and pass.

Follow-up, not in this PR

Once this is green, the three checks are candidates for required status checks on main — which is the point of having them.

🤖 Generated with Claude Code

This repository has no CI. `gh pr checks` reports zero checks on every open PR,
so the four fixture suites run only when someone remembers to run them.

That is a poor fit for what they guard. Every defect these suites pin fails in
the same direction -- reporting an infected file or repository as CLEAN -- so a
regression here does not break a build, it quietly stops finding malware. The
repo has already been bitten by exactly this: test_config_classification.sh had
no assertions at all and passed for weeks while testing nothing, which is
recorded in f5bc89e.

Suites are DISCOVERED (test_*.sh at the root), not listed. Listing them means a
new suite silently never runs until someone edits this file, and it also breaks
the other way: test_org_sweep.sh does not exist on main yet -- it arrives with
PR #3 -- so a workflow naming it would fail on main for the wrong reason. The
job errors out if it discovers zero suites, so "passed by finding nothing"
cannot happen either.

Runs on a matrix of both search engines. The scanner falls back to grep where no
ripgrep BINARY exists, because `rg` is frequently only a shell function and
`command -v rg` can succeed in an interactive zsh while failing inside the
script. That exact difference hid a real detection failure, so the grep path
needs to be covered rather than assumed.

Also runs bash -n over every script. Syntax only: these are bash-specific and
macOS bash-3.2 compatible on purpose, so a style linter would fight the code.

actions/checkout is pinned by SHA, verified to resolve to v5.0.0.

Verified locally: the three suites on main are discovered and pass.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@fabrizzio-dotCMS

Copy link
Copy Markdown
Member Author

Folded into #3 — commit 34c571b is now part of that branch, and the suites it runs are in the same PR.

@fabrizzio-dotCMS
fabrizzio-dotCMS deleted the ci/run-test-suites branch September 1, 2026 20:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant