ci(control-tests): run the suite against both auditor images#68
Merged
stevebeattie merged 1 commit intoJun 18, 2026
Merged
Conversation
Why: the Chainguard cinc-auditor image (now the primary, per the default switch) and the upstream cincproject/auditor image are built independently and can diverge — e.g. the nonroot-vs-root default user and the /usr/lib-vs-/opt bundle layout already surfaced in cinc-check. The per-PR control-tests gate should catch a profile regression on either image, not just the primary. What: matrix the control-tests `test` job over both images (chainguard primary + cincproject), fail-fast: false so one image's failure doesn't mask the other; the job name shows which image, and CINC_AUDITOR_IMAGE is taken from the matrix. The tooling is already image-default-agnostic (every docker invocation forces --user 0:0; cinc-check scans an empty rootfs), so both legs run unchanged. scan-smoke stays single-image: it exercises the scan pipeline (export/overlay/ live/find), which is image-agnostic, so doubling its 8-entry matrix adds little signal for a lot of weekly cost. Verified both legs locally: make cinc-check and a control spec pass on each of cgr.dev/chainguard/cinc-auditor:latest and cincproject/auditor:latest (the full chainguard suite is 186 examples, 0 failures). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
After #67 made the public Chainguard cinc-auditor image the default, the
per-PR control-tests gate runs against only one image. The Chainguard image and
the upstream
cincproject/auditorimage are built independently and can diverge— the nonroot-vs-root default user and the
/usr/lib-vs-/optbundle layoutalready surfaced in cinc-check (#67). The gate should catch a profile regression
on either image, not just the primary.
What
Matrix the control-tests
testjob over both images:cgr.dev/chainguard/cinc-auditor:latest(primary)cincproject/auditor:latest(upstream)fail-fast: falseso one image's failure doesn't mask the other; the job nameshows which image (
test (chainguard)/test (cincproject)), andCINC_AUDITOR_IMAGEis taken from the matrix (a static workflow literal — nountrusted input).
The tooling is already image-default-agnostic after #67 (every docker
invocation forces
--user 0:0; cinc-check scans an empty rootfs), so both legsrun unchanged.
scan-smokestays single-image deliberately: it exercises the scan pipeline(export / overlay / live / find), which is image-agnostic, so doubling its
8-entry matrix adds little signal for a lot of weekly cost.
Verification
Both legs verified locally:
make cinc-checkand a control spec pass on each ofcgr.dev/chainguard/cinc-auditor:latestandcincproject/auditor:latest(thefull chainguard suite is 186 examples, 0 failures). actionlint + zizmor clean.
🤖 Generated with Claude Code