Skip to content

feat(ci): build must-gather image hermetically in upstream repo [RHIDP-14412] - #218

Open
Fortune-Ndlovu wants to merge 7 commits into
redhat-developer:mainfrom
Fortune-Ndlovu:RHIDP-14412-build-must-gather-image-hermetically-in-upstream-repo
Open

feat(ci): build must-gather image hermetically in upstream repo [RHIDP-14412]#218
Fortune-Ndlovu wants to merge 7 commits into
redhat-developer:mainfrom
Fortune-Ndlovu:RHIDP-14412-build-must-gather-image-hermetically-in-upstream-repo

Conversation

@Fortune-Ndlovu

@Fortune-Ndlovu Fortune-Ndlovu commented Jul 6, 2026

Copy link
Copy Markdown
Member

Description

This PR replaces the non-hermetic Containerfile build with a Hermeto-based hermetic build pipeline, following the same pattern used by the RHDH upstream repo. This catches hermetic build issues earlier (before downstream Konflux) and validates Renovate PRs that update hermetic-only dependencies like rpms.lock.yaml.

Which issue(s) does this PR fix or relate to

PR acceptance criteria

  • Tests
  • Documentation

How to test changes / Special notes to the reviewer

To test the hermetic build locally, we can replicate what the composite action does by running Hermeto to prefetch deps, then buildah with --network=none

  # 1. Copy requirements to root (matches what the composite action does)
  cp .rhdh/docker/requirements*.txt .

  # 2. Prefetch all dependencies with Hermeto
  HERMETO_IMAGE="quay.io/konflux-ci/hermeto@sha256:de5398035bee2965a275313654f51c80e213ae460b6ec5a23e32c81efd00e67b"
  CACHE_DIR="./hermeto-cache/rhdh-must-gather"
  mkdir -p "$CACHE_DIR"

  podman run --rm -v "$PWD:/source:z" -v "$CACHE_DIR:/cachi2:z" -w /source "$HERMETO_IMAGE" \
    --log-level DEBUG fetch-deps --dev-package-managers \
    --source . --output /cachi2/output \
    '[{"type": "rpm", "path": "."}, {"type": "pip", "path": ".", "allow_binary": "false"}, {"type": "generic", "path": "."}, {"type": "cargo", "path": 
  "vendor/websocat"}]'

  # 3. Generate env + inject files
  podman run --rm -v "$PWD:/source:z" -v "$CACHE_DIR:/cachi2:z" -w /source "$HERMETO_IMAGE" \
    --log-level DEBUG generate-env --format env --output /cachi2/cachi2.env /cachi2/output

  podman run --rm -v "$PWD:/source:z" -v "$CACHE_DIR:/cachi2:z" -w /source "$HERMETO_IMAGE" \
    --log-level DEBUG inject-files /cachi2/output

  # 4. Build hermetically
  EMPTY_DIR=$(mktemp -d)
  podman build --network=none \
    --volume "$(realpath $CACHE_DIR):/cachi2:z" \
    --volume "$EMPTY_DIR:/run/secrets/etc-pki-entitlement:z" \
    -f .rhdh/docker/Containerfile \
    -t rhdh-must-gather:test .
  rm -rf "$EMPTY_DIR"

  # 5. Smoke test
  mkdir -p /tmp/mg-test
  podman run --rm -v /tmp/mg-test:/must-gather:z rhdh-must-gather:test --help

or using pr image

  # OpenShift (with oc):
  oc adm must-gather --image=localhost/rhdh-must-gather:pr-218

  # Kubernetes (with Kind/minikube — uses Helm chart):
  make deploy-k8s FULL_IMAGE_NAME=localhost/rhdh-must-gather:pr-218

  # E2E tests (spins up Kind cluster + runs validation):
  make test-e2e LOCAL=false FULL_IMAGE_NAME=localhost/rhdh-must-gather:pr-218

  # Quick local test (no container, just scripts against current kubeconfig):
  make run-local

@openshift-ci
openshift-ci Bot requested review from rm3l and subhashkhileri July 6, 2026 00:37
@rhdh-qodo-merge

Copy link
Copy Markdown

PR Summary by Qodo

Hermetic CI build for must-gather image via Hermeto + locked dependencies

✨ Enhancement ⚙️ Configuration changes 🕐 40+ Minutes

Grey Divider

AI Description

• Build the must-gather image in upstream CI using Hermeto + network-isolated Buildah.
• Replace non-reproducible downloads/installs with locked RPM, pip-hash, and Cargo inputs.
• Add automation and tooling for maintaining RPM/Python lock artifacts and local hermetic builds.
Diagram

graph TD
  A["PR Build workflow"] --> B[["docker-build action"]] --> C{{"Hermeto"}} --> D["cachi2 cache + env"] --> E["Containerfile.hermeto"] --> F["Buildah build (--network=none)"]
  G["Container build workflow"] --> B
  H["Local Makefile/script"] --> C
  F --> I["Image artifact / registry"]

  subgraph Legend
    direction LR
    _wf["Workflow"] ~~~ _act[["Composite action"]] ~~~ _ext{{"External tool"}}
  end
Loading
High-Level Assessment

The following are alternative approaches to this PR:

1. Reusable workflow (workflow_call) instead of composite action
  • ➕ Centralizes permissions/runner setup once (less duplication across repos)
  • ➕ Easier to version and consume across multiple repositories
  • ➖ Harder to run locally than a composite action + script
  • ➖ More YAML indirection for reviewers/debugging
2. Use BuildKit/buildx for hermetic builds
  • ➕ Potentially faster builds with better caching primitives
  • ➕ More common in GitHub-hosted CI environments
  • ➖ May diverge from Konflux/Buildah parity the PR is explicitly targeting
  • ➖ Would require reworking signing/push steps and cache mounts
3. Rely solely on downstream Konflux for hermetic validation
  • ➕ Avoids duplicating hermetic build logic in upstream CI
  • ➕ Less GitHub Actions maintenance
  • ➖ Detects hermetic breakage later (after merge)
  • ➖ Renovate updates to hermetic-only inputs (e.g., rpms.lock.yaml) wouldn’t be validated upstream

Recommendation: Keep the current approach: it intentionally mirrors downstream Konflux (Hermeto + cachi2 + network-isolated build) while providing an upstream safety net and a local reproduction script. If this pattern needs to be shared across multiple repos, consider migrating the composite action into a reusable workflow later.

Files changed (17) +775 / -197

Documentation (1) +2 / -2
pre-commit-comment.yamlUpdate pre-commit helper text for root requirements paths +2/-2

Update pre-commit helper text for root requirements paths

• Fixes documentation in the bot comment to reference requirements.in/requirements.txt at repo root instead of the prior .rhdh/docker paths.

.github/workflows/pre-commit-comment.yaml

Other (16) +773 / -195
.containerignoreIgnore Hermeto cache artifacts in build context +4/-0

Ignore Hermeto cache artifacts in build context

• Adds hermeto-cache/ and transformed Containerfile.hermeto to the container build ignore list to reduce context size and avoid accidental inclusion.

.containerignore

action.yamlAdd composite action to build images hermetically +189/-0

Add composite action to build images hermetically

• Introduces a composite GitHub Action that runs Hermeto to fetch RPM/pip/cargo deps, generates cachi2.env, injects files, transforms the Containerfile, and builds via buildah-build with --network=none. Optionally saves the built image as a workflow artifact.

.github/actions/docker-build/action.yaml

renovate.jsonAlign Renovate pip-compile scanning to root requirements files +1/-28

Align Renovate pip-compile scanning to root requirements files

• Updates pip-compile manager patterns to track requirements*.txt at repo root and removes custom datasources/managers that were only needed for curl-based oc/helm downloads.

.github/renovate.json

container-build.yamlSwitch release/main image publish workflow to hermetic build action +44/-48

Switch release/main image publish workflow to hermetic build action

• Replaces Makefile-based build/push with the new hermetic composite action, then pushes tags/digests and signs using cosign. Updates tagging, digest capture, and summary text for hermetic builds.

.github/workflows/container-build.yaml

pr-build.yamlBuild PR images hermetically and standardize artifacts +35/-44

Build PR images hermetically and standardize artifacts

• Uses the composite hermetic build action when relevant files change, expands the changed-files filter to include lock/requirements inputs, and adjusts artifact naming to podman-image-<PR>. Updates e2e job to load the new image.tar artifact.

.github/workflows/pr-build.yaml

pr-push.yamlUpdate PR push workflow for new artifact naming and hermetic build job name +33/-20

Update PR push workflow for new artifact naming and hermetic build job name

• Tracks the renamed PR Build workflow, downloads the new per-PR image artifact plus metadata artifact, and continues to validate metadata to mitigate artifact poisoning. Updates load/push steps to use image.tar and improves messaging when artifacts are absent.

.github/workflows/pr-push.yaml

update-rpm-lockfile.yamlAdd scheduled workflow to refresh rpms.lock.yaml via rpm-lockfile-prototype +125/-0

Add scheduled workflow to refresh rpms.lock.yaml via rpm-lockfile-prototype

• Adds a weekly (and on-demand) workflow that runs rpm-lockfile-prototype and creates a PR when rpms.lock.yaml changes, including auto-labeling and auto-approval comments using a bot token.

.github/workflows/update-rpm-lockfile.yaml

.pre-commit-config.yamlMove pip-compile/uv hooks to root requirements files +9/-9

Move pip-compile/uv hooks to root requirements files

• Updates hook file matchers and output paths to generate requirements.txt and requirements-build.txt at repo root and keeps the YQ version check aligned with requirements.in.

.pre-commit-config.yaml

ContainerfileMake image build reproducible (locked Cargo, RPM CLIs, hashed pip installs) +13/-35

Make image build reproducible (locked Cargo, RPM CLIs, hashed pip installs)

• Adds --locked to the vendored websocat cargo build for reproducibility. Replaces curl-downloaded oc/kubectl and helm with RPM-installed openshift-clients and helm, and installs Python deps from pinned requirements files using --require-hashes.

Containerfile

MakefileRoute image builds through local Hermeto script and add cache target +11/-3

Route image builds through local Hermeto script and add cache target

• Replaces image-build with a hermetic build invocation and introduces a hermeto-cache target for dependency-only caching. Extends clean to remove hermeto-cache and the transformed Containerfile.hermeto artifact.

Makefile

check-requirements-version.shUpdate YQ requirements version check to new requirements.in location +1/-1

Update YQ requirements version check to new requirements.in location

• Points the pre-commit version check script at requirements.in (repo root) instead of .rhdh/docker/requirements.in.

hack/check-requirements-version.sh

requirements-build.inAdd pinned build-backend dependency input for hermetic pip builds +0/-0

Add pinned build-backend dependency input for hermetic pip builds

• Defines the set of PEP 517 build backend dependencies required to build sdists under hermetic/network-isolated conditions (targeting Python 3.9 on RHEL 9).

requirements-build.in

requirements-build.txtRegenerate build dependency lockfile with hashes +5/-5

Regenerate build dependency lockfile with hashes

• Updates the generated, hashed lockfile for build backends and adjusts header comments to match the new root file locations and uv command.

requirements-build.txt

requirements.inAdd root requirements input for yq pinning +0/-0

Add root requirements input for yq pinning

• Provides the pinned yq version input file used by pip-compile and the Makefile/version-check hook.

requirements.in

requirements.txtUpdate pip-compile output path metadata for root requirements +2/-2

Update pip-compile output path metadata for root requirements

• Adjusts generated header/comments for the requirements lockfile and aligns references to requirements.in at repo root (content remains hash-pinned).

requirements.txt

local-hermeto-build.shAdd local Hermeto build simulator for cache + hermetic image builds +301/-0

Add local Hermeto build simulator for cache + hermetic image builds

• Adds a local script that mirrors Konflux Hermeto behavior: fetch-deps/generate-env/inject-files, transforms Containerfile to use cachi2 RPM repos and env injection, and builds with podman using --network none. Supports cross-platform builds via TARGET_PLATFORM and includes macOS GNU sed checks.

scripts/local-hermeto-build.sh

@rhdh-qodo-merge

rhdh-qodo-merge Bot commented Jul 6, 2026

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

Grey Divider


Action required

1. Version arg not passed ✓ Resolved 🐞 Bug ≡ Correctness
Description
The hermetic build action builds the Containerfile without passing `--build-arg
RHDH_MUST_GATHER_VERSION=..., so images bake the default 0.0.0-unknown` into LABEL/ENV and
collection-scripts/version will report the wrong version. This is a regression from the previous
Makefile-driven build that set the build arg explicitly.
Code

.github/actions/docker-build/action.yaml[R152-164]

+    - name: "Build Docker Image"
+      id: build
+      uses: redhat-actions/buildah-build@7a95fa7ee0f02d552a32753e7414641a04307056 # v2.13
+      with:
+        containerfiles: ${{ steps.transform-containerfile.outputs.transformed_containerfile }}
+        context: .
+        platform: ${{ inputs.platform }}
+        tags: ${{ steps.meta.outputs.tags }}
+        labels: ${{ steps.meta.outputs.labels }}
+        extra-args: |
+          --network=none
+          --volume ${{ env.LOCAL_CACHE_DIR_REALPATH }}:/cachi2:z
+
Relevance

●● Moderate

Mixed history: PRs used version build-arg, but reviewer rejected changing default version ARG/ENV
semantics.

PR-#109
PR-#137

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The Containerfile’s version is derived from RHDH_MUST_GATHER_VERSION, but the new hermetic build
invocations never set the build arg, so the default 0.0.0-unknown becomes the baked image version
and is what the version script prints.

Containerfile[19-27]
Containerfile[72-73]
collection-scripts/version[5-10]
.github/actions/docker-build/action.yaml[152-164]
scripts/local-hermeto-build.sh[200-209]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The new hermetic build path never sets `RHDH_MUST_GATHER_VERSION` at build time. As a result, the image LABEL/ENV remains the Containerfile default (`0.0.0-unknown`) and `collection-scripts/version` will always print that incorrect value.

## Issue Context
`Containerfile` uses `ARG RHDH_MUST_GATHER_VERSION` to populate both LABEL and ENV. The composite build action uses `buildah-build` but does not provide any build args, and the local hermeto script similarly calls `podman build` without `--build-arg`.

## Fix Focus Areas
- .github/actions/docker-build/action.yaml[152-164]
- .github/workflows/container-build.yaml[87-96]
- .github/workflows/pr-build.yaml[79-88]
- scripts/local-hermeto-build.sh[200-209]
- Makefile[195-199]

## What to change
- Extend the composite action to accept a version/build-arg input (e.g. `mustGatherVersion`) and pass it through to the build command (either via a supported `build-args:` input on `redhat-actions/buildah-build` or by adding `--build-arg RHDH_MUST_GATHER_VERSION=...` to `extra-args`).
- Update `container-build.yaml` and `pr-build.yaml` to pass the computed version (e.g. `${{ steps.prep.outputs.VERSION }}`) into that new input.
- Update `scripts/local-hermeto-build.sh` (and the `Makefile` target) to pass `--build-arg RHDH_MUST_GATHER_VERSION=...` so `make image-build` behaves consistently.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools



Remediation recommended

2. Floating Hermeto image tag ✓ Resolved 🐞 Bug ⛨ Security
Description
Both the CI composite action and the local hermeto script use quay.io/konflux-ci/hermeto:latest,
so the supposedly hermetic pipeline is not reproducible and can change behavior or break without any
change in this repo. This also increases supply-chain risk because the build tool image is not
pinned to an immutable digest.
Code

.github/actions/docker-build/action.yaml[R64-69]

+    - name: Set up hermetic build variables
+      shell: bash
+      run: |
+        echo "HERMETO_IMAGE=quay.io/konflux-ci/hermeto:latest" >> $GITHUB_ENV
+        echo "LOCAL_CACHE_DIR=./hermeto-cache/rhdh-must-gather" >> $GITHUB_ENV
+
Relevance

●●● Strong

Team previously accepted removing floating :latest tags for reproducibility (pin UBI tags/digests).

PR-#137
PR-#181

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The CI action sets HERMETO_IMAGE to a :latest tag, and the local build script hardcodes the same
tag, making the toolchain mutable across time.

.github/actions/docker-build/action.yaml[64-69]
scripts/local-hermeto-build.sh[25-27]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Using the floating `:latest` tag for Hermeto makes builds non-deterministic and subject to upstream drift.

## Issue Context
Hermeto is the core dependency-fetch and injection tool for this pipeline; changes in the tool image affect the produced build inputs.

## Fix Focus Areas
- .github/actions/docker-build/action.yaml[64-69]
- scripts/local-hermeto-build.sh[25-27]

## What to change
- Replace `quay.io/konflux-ci/hermeto:latest` with an immutable reference (preferably `@sha256:<digest>`).
- Consider defining the pinned image reference once (e.g., repo-level env or shared script constant) and reusing it in both CI and `scripts/local-hermeto-build.sh`.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


3. Unpinned lockfile tool source ✓ Resolved 🐞 Bug ⛨ Security
Description
The new RPM lockfile workflow installs rpm-lockfile-prototype from refs/heads/main.zip, which
executes unpinned upstream code and can change behavior at any time. This makes automated lockfile
updates non-reproducible and increases supply-chain risk for a workflow that runs on schedule.
Code

.github/workflows/update-rpm-lockfile.yaml[R39-55]

+      - name: Install rpm-lockfile-prototype
+        run: |
+          if [[ ! -x "${HOME}/.local/bin/rpm-lockfile-prototype" ]]; then
+            echo "Installing rpm-lockfile-prototype ..."
+
+            sudo apt-get update
+            sudo apt-get install -y python3 python3-pip python3-dev build-essential
+            sudo apt-get install -y podman skopeo rpm
+            sudo apt-get install -y dnf python3-dnf
+
+            mkdir -p "${HOME}/.local/bin/"
+            python3 -m pip install --user https://github.com/konflux-ci/rpm-lockfile-prototype/archive/refs/heads/main.zip
+
+            # Update PATH
+            export PATH=${PATH%":${HOME}/.local/bin"}:${HOME}/.local/bin
+            echo "${HOME}/.local/bin" >> $GITHUB_PATH
+          else
Relevance

●●● Strong

Repo strongly prefers immutable/pinned refs in CI for supply-chain hygiene (many digest-pinned
action updates).

PR-#10
PR-#181
PR-#143

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The workflow explicitly installs from refs/heads/main.zip, which is not stable across runs and can
change without changes to this repository.

.github/workflows/update-rpm-lockfile.yaml[39-55]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The workflow installs `rpm-lockfile-prototype` from the upstream `main` branch ZIP, which is a moving target.

## Issue Context
This job runs on a schedule and can open PRs automatically; changes in the upstream `main` branch can silently change generated output or break the workflow.

## Fix Focus Areas
- .github/workflows/update-rpm-lockfile.yaml[39-55]

## What to change
- Pin the installation to an immutable reference:
 - prefer a tagged release if available, or
 - pin to a specific commit SHA archive URL, or
 - vendor/package the tool in a controlled way.
- Optionally, log the installed tool version/commit in the workflow summary for traceability.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


4. Hermeto failures partially masked ✓ Resolved 🐞 Bug ☼ Reliability
Description
The composite action suppresses errors from hermeto fetch-deps and hermeto inject-files using
|| echo, so the step may proceed even when Hermeto partially fails and leaves stale/invalid output
in the cache directory. This can create flaky or non-hermetic builds and makes the job logs
misleading (success path despite failed dependency caching).
Code

.github/actions/docker-build/action.yaml[R70-105]

+    - name: Cache dependencies with hermeto
+      shell: bash
+      run: |
+        set -ex
+
+        echo "::group::Creating local cache directory"
+        mkdir -p ${{ env.LOCAL_CACHE_DIR }} || echo "Failed to create local cache directory"
+        echo "::endgroup::"
+
+        echo "::group::Fetching dependencies with hermeto"
+        podman run --rm -v "$PWD:/source:z" -v "$LOCAL_CACHE_DIR:/cachi2:z" -w /source "$HERMETO_IMAGE" \
+          --log-level DEBUG \
+          fetch-deps --dev-package-managers \
+          --source . \
+          --output /cachi2/output \
+          '[{"type": "rpm", "path": "."}, {"type": "pip", "path": ".", "allow_binary": "false"}, {"type": "cargo", "path": "vendor/websocat"}]' || echo "Fetch-deps failed"
+        echo "::endgroup::"
+
+        if [ -d ${{ env.LOCAL_CACHE_DIR }}/output ]; then
+          echo "::group::Generating environment file"
+          podman run --rm -v "$PWD:/source:z" -v "$LOCAL_CACHE_DIR:/cachi2:z" -w /source "$HERMETO_IMAGE" \
+                      --log-level DEBUG \
+                      generate-env --format env \
+                      --output /cachi2/cachi2.env /cachi2/output
+          echo "::endgroup::"
+        else
+          echo "No output directory found, skipping generate-env"
+          exit 1
+        fi
+
+        if [ -d ${{ env.LOCAL_CACHE_DIR }}/output ]; then
+          echo "::group::Injecting files"
+          podman run --rm -v "$PWD:/source:z" -v "$LOCAL_CACHE_DIR:/cachi2:z" -w /source "$HERMETO_IMAGE" \
+            --log-level DEBUG \
+            inject-files /cachi2/output || echo "Inject-files failed"
+          echo "::endgroup::"
Relevance

●● Moderate

No clear prior pattern on accepting/rejecting masking failures with || echo in CI steps.

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The action explicitly appends || echo ... to Hermeto commands, which neutralizes failures even
under set -e, so the workflow can continue after dependency caching errors as long as the output
directory exists.

.github/actions/docker-build/action.yaml[70-105]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The dependency-caching step masks Hermeto failures by converting non-zero exits into success via `|| echo ...`. This can allow stale or incomplete cache artifacts to be used later, producing unreliable build results.

## Issue Context
The step runs with `set -e`, but `cmd || echo ...` forces a zero exit status. The only guard is the existence of the output directory, not correctness/completeness of the generated env/deps.

## Fix Focus Areas
- .github/actions/docker-build/action.yaml[70-111]

## What to change
- Remove `|| echo "Fetch-deps failed"` and `|| echo "Inject-files failed"` so failures fail the workflow.
- Optionally, before fetching deps, delete/clean `${LOCAL_CACHE_DIR}/output` to avoid reusing stale output from previous attempts.
- Consider validating required artifacts exist (e.g., `/cachi2/output/deps/...` and `/cachi2/cachi2.env`) before proceeding to the build step.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools



Informational

5. Stale requirements path comments ✓ Resolved 🐞 Bug ⚙ Maintainability
Description
requirements-build.txt still contains autogenerated “via” comments referencing the old
.rhdh/docker/requirements-build.in path, which is misleading now that requirements files live at
the repo root. This won’t break builds but will confuse future updates/regeneration.
Code

requirements-build.txt[R1-21]

# This file was autogenerated by uv via the following command:
-#    uv pip compile --python-version=3.9 --python-platform=linux --generate-hashes --output-file=.rhdh/docker/requirements-build.txt .rhdh/docker/requirements-build.in
+#    uv pip compile --python-version=3.9 --python-platform=linux --generate-hashes --output-file=requirements-build.txt requirements-build.in
calver==2025.10.20 \
    --hash=sha256:c98b376c2424642224d456b2f70c51402343e008c63d204634665e1a2a2835f5 \
    --hash=sha256:d7d75224eed9d9263f4fb30008487615196d208d14752bfd93fea7af2c84f508
-    # via -r .rhdh/docker/requirements-build.in
+    # via -r requirements-build.in
editables==0.6 \
    --hash=sha256:1163834902381c4613787951c5914800fdf155ae08848a373b8ea5006780977c \
    --hash=sha256:d70e4698078a1d033e7786d9c64e5be070d058a67c21417024d38a58ac20aa43
    # via hatchling
flit-core==3.12.0 \
    --hash=sha256:18f63100d6f94385c6ed57a72073443e1a71a4acb4339491615d0f16d6ff01b2 \
    --hash=sha256:e7a0304069ea895172e3c7bb703292e992c5d1555dd1233ab7b5621b5b69e62c
-    # via -r .rhdh/docker/requirements-build.in
+    # via -r requirements-build.in
hatch-vcs==0.4.0 \
    --hash=sha256:093810748fe01db0d451fabcf2c1ac2688caefd232d4ede967090b1c1b07d9f7 \
    --hash=sha256:b8a2b6bee54cf6f9fc93762db73890017ae59c9081d1038a41f16235ceaf8b2c
-    # via -r .rhdh/docker/requirements-build.in
+    # via -r requirements-build.in
hatchling==1.21.1 \
    --hash=sha256:21e8c13f8458b219a91cb84e5b61c15bf786695d1c4fabc29e91e78f94bfe892 \
    --hash=sha256:bba440453a224e7d4478457fa2e8d8c3633765bafa02975a6b53b9bf917980bc
Relevance

●●● Strong

Repo often accepts small maintainability/typo fixes and requirements tooling cleanups.

PR-#57
PR-#144

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The file header indicates regeneration from requirements-build.in, but multiple # via lines
still reference .rhdh/docker/requirements-build.in, which no longer matches the repo layout.

requirements-build.txt[1-6]
requirements-build.txt[19-56]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Some autogenerated `# via -r ...` comments still reference the removed `.rhdh/docker/...` path.

## Issue Context
The file header was updated to root paths, but not all embedded comments were updated.

## Fix Focus Areas
- requirements-build.txt[19-56]

## What to change
- Regenerate `requirements-build.txt` using the documented command in `requirements-build.in`, or manually update the stale comment lines to `requirements-build.in` so the file is internally consistent.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

Context
⚠️ Tickets: not configured — ticket URL found in PR but could not be fetched — check ticket provider credentials
✅ Compliance rules (platform): 7 rules
✅ Cross-repo context
  Not relevant to this PR: redhat-developer/rhdh-chart

Grey Divider

Tip of the day
💡 Did you know, you can describe a rule in plain language on the Rules page and Qodo drafts it for you

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Grey Divider

Qodo Logo

Comment thread .github/actions/docker-build/action.yaml
Comment thread .github/actions/docker-build/action.yaml
Comment thread .github/actions/docker-build/action.yaml
Comment thread .github/workflows/update-rpm-lockfile.yaml
Comment thread requirements-build.txt
@Fortune-Ndlovu

Copy link
Copy Markdown
Member Author

/agentic_review

Comment thread .github/actions/docker-build/action.yaml Outdated
Comment thread Containerfile Outdated
@Fortune-Ndlovu

Copy link
Copy Markdown
Member Author

FYI I will be resolving conflicts soon....

@Fortune-Ndlovu
Fortune-Ndlovu force-pushed the RHIDP-14412-build-must-gather-image-hermetically-in-upstream-repo branch from 0759b41 to 8fca11e Compare August 14, 2026 14:12
@Fortune-Ndlovu

Copy link
Copy Markdown
Member Author

/agentic_review

@rhdh-qodo-merge

rhdh-qodo-merge Bot commented Aug 14, 2026

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

Grey Divider


Action required

1. Version arg not passed ✓ Resolved 🐞 Bug ≡ Correctness
Description
The hermetic build action builds the Containerfile without passing `--build-arg
RHDH_MUST_GATHER_VERSION=..., so images bake the default 0.0.0-unknown` into LABEL/ENV and
collection-scripts/version will report the wrong version. This is a regression from the previous
Makefile-driven build that set the build arg explicitly.
Code

.github/actions/docker-build/action.yaml[R152-164]

+    - name: "Build Docker Image"
+      id: build
+      uses: redhat-actions/buildah-build@7a95fa7ee0f02d552a32753e7414641a04307056 # v2.13
+      with:
+        containerfiles: ${{ steps.transform-containerfile.outputs.transformed_containerfile }}
+        context: .
+        platform: ${{ inputs.platform }}
+        tags: ${{ steps.meta.outputs.tags }}
+        labels: ${{ steps.meta.outputs.labels }}
+        extra-args: |
+          --network=none
+          --volume ${{ env.LOCAL_CACHE_DIR_REALPATH }}:/cachi2:z
+
Relevance

●● Moderate

Mixed history: PRs used version build-arg, but reviewer rejected changing default version ARG/ENV
semantics.

PR-#109
PR-#137

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The Containerfile’s version is derived from RHDH_MUST_GATHER_VERSION, but the new hermetic build
invocations never set the build arg, so the default 0.0.0-unknown becomes the baked image version
and is what the version script prints.

Containerfile[19-27]
Containerfile[72-73]
collection-scripts/version[5-10]
.github/actions/docker-build/action.yaml[152-164]
scripts/local-hermeto-build.sh[200-209]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The new hermetic build path never sets `RHDH_MUST_GATHER_VERSION` at build time. As a result, the image LABEL/ENV remains the Containerfile default (`0.0.0-unknown`) and `collection-scripts/version` will always print that incorrect value.

## Issue Context
`Containerfile` uses `ARG RHDH_MUST_GATHER_VERSION` to populate both LABEL and ENV. The composite build action uses `buildah-build` but does not provide any build args, and the local hermeto script similarly calls `podman build` without `--build-arg`.

## Fix Focus Areas
- .github/actions/docker-build/action.yaml[152-164]
- .github/workflows/container-build.yaml[87-96]
- .github/workflows/pr-build.yaml[79-88]
- scripts/local-hermeto-build.sh[200-209]
- Makefile[195-199]

## What to change
- Extend the composite action to accept a version/build-arg input (e.g. `mustGatherVersion`) and pass it through to the build command (either via a supported `build-args:` input on `redhat-actions/buildah-build` or by adding `--build-arg RHDH_MUST_GATHER_VERSION=...` to `extra-args`).
- Update `container-build.yaml` and `pr-build.yaml` to pass the computed version (e.g. `${{ steps.prep.outputs.VERSION }}`) into that new input.
- Update `scripts/local-hermeto-build.sh` (and the `Makefile` target) to pass `--build-arg RHDH_MUST_GATHER_VERSION=...` so `make image-build` behaves consistently.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools



Remediation recommended

2. Floating Hermeto image tag ✓ Resolved 🐞 Bug ⛨ Security
Description
Both the CI composite action and the local hermeto script use quay.io/konflux-ci/hermeto:latest,
so the supposedly hermetic pipeline is not reproducible and can change behavior or break without any
change in this repo. This also increases supply-chain risk because the build tool image is not
pinned to an immutable digest.
Code

.github/actions/docker-build/action.yaml[R64-69]

+    - name: Set up hermetic build variables
+      shell: bash
+      run: |
+        echo "HERMETO_IMAGE=quay.io/konflux-ci/hermeto:latest" >> $GITHUB_ENV
+        echo "LOCAL_CACHE_DIR=./hermeto-cache/rhdh-must-gather" >> $GITHUB_ENV
+
Relevance

●●● Strong

Team previously accepted removing floating :latest tags for reproducibility (pin UBI tags/digests).

PR-#137
PR-#181

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The CI action sets HERMETO_IMAGE to a :latest tag, and the local build script hardcodes the same
tag, making the toolchain mutable across time.

.github/actions/docker-build/action.yaml[64-69]
scripts/local-hermeto-build.sh[25-27]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Using the floating `:latest` tag for Hermeto makes builds non-deterministic and subject to upstream drift.

## Issue Context
Hermeto is the core dependency-fetch and injection tool for this pipeline; changes in the tool image affect the produced build inputs.

## Fix Focus Areas
- .github/actions/docker-build/action.yaml[64-69]
- scripts/local-hermeto-build.sh[25-27]

## What to change
- Replace `quay.io/konflux-ci/hermeto:latest` with an immutable reference (preferably `@sha256:<digest>`).
- Consider defining the pinned image reference once (e.g., repo-level env or shared script constant) and reusing it in both CI and `scripts/local-hermeto-build.sh`.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


3. Unpinned lockfile tool source ✓ Resolved 🐞 Bug ⛨ Security
Description
The new RPM lockfile workflow installs rpm-lockfile-prototype from refs/heads/main.zip, which
executes unpinned upstream code and can change behavior at any time. This makes automated lockfile
updates non-reproducible and increases supply-chain risk for a workflow that runs on schedule.
Code

.github/workflows/update-rpm-lockfile.yaml[R39-55]

+      - name: Install rpm-lockfile-prototype
+        run: |
+          if [[ ! -x "${HOME}/.local/bin/rpm-lockfile-prototype" ]]; then
+            echo "Installing rpm-lockfile-prototype ..."
+
+            sudo apt-get update
+            sudo apt-get install -y python3 python3-pip python3-dev build-essential
+            sudo apt-get install -y podman skopeo rpm
+            sudo apt-get install -y dnf python3-dnf
+
+            mkdir -p "${HOME}/.local/bin/"
+            python3 -m pip install --user https://github.com/konflux-ci/rpm-lockfile-prototype/archive/refs/heads/main.zip
+
+            # Update PATH
+            export PATH=${PATH%":${HOME}/.local/bin"}:${HOME}/.local/bin
+            echo "${HOME}/.local/bin" >> $GITHUB_PATH
+          else
Relevance

●●● Strong

Repo strongly prefers immutable/pinned refs in CI for supply-chain hygiene (many digest-pinned
action updates).

PR-#10
PR-#181
PR-#143

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The workflow explicitly installs from refs/heads/main.zip, which is not stable across runs and can
change without changes to this repository.

.github/workflows/update-rpm-lockfile.yaml[39-55]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The workflow installs `rpm-lockfile-prototype` from the upstream `main` branch ZIP, which is a moving target.

## Issue Context
This job runs on a schedule and can open PRs automatically; changes in the upstream `main` branch can silently change generated output or break the workflow.

## Fix Focus Areas
- .github/workflows/update-rpm-lockfile.yaml[39-55]

## What to change
- Pin the installation to an immutable reference:
 - prefer a tagged release if available, or
 - pin to a specific commit SHA archive URL, or
 - vendor/package the tool in a controlled way.
- Optionally, log the installed tool version/commit in the workflow summary for traceability.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


4. Hermeto failures partially masked ✓ Resolved 🐞 Bug ☼ Reliability
Description
The composite action suppresses errors from hermeto fetch-deps and hermeto inject-files using
|| echo, so the step may proceed even when Hermeto partially fails and leaves stale/invalid output
in the cache directory. This can create flaky or non-hermetic builds and makes the job logs
misleading (success path despite failed dependency caching).
Code

.github/actions/docker-build/action.yaml[R70-105]

+    - name: Cache dependencies with hermeto
+      shell: bash
+      run: |
+        set -ex
+
+        echo "::group::Creating local cache directory"
+        mkdir -p ${{ env.LOCAL_CACHE_DIR }} || echo "Failed to create local cache directory"
+        echo "::endgroup::"
+
+        echo "::group::Fetching dependencies with hermeto"
+        podman run --rm -v "$PWD:/source:z" -v "$LOCAL_CACHE_DIR:/cachi2:z" -w /source "$HERMETO_IMAGE" \
+          --log-level DEBUG \
+          fetch-deps --dev-package-managers \
+          --source . \
+          --output /cachi2/output \
+          '[{"type": "rpm", "path": "."}, {"type": "pip", "path": ".", "allow_binary": "false"}, {"type": "cargo", "path": "vendor/websocat"}]' || echo "Fetch-deps failed"
+        echo "::endgroup::"
+
+        if [ -d ${{ env.LOCAL_CACHE_DIR }}/output ]; then
+          echo "::group::Generating environment file"
+          podman run --rm -v "$PWD:/source:z" -v "$LOCAL_CACHE_DIR:/cachi2:z" -w /source "$HERMETO_IMAGE" \
+                      --log-level DEBUG \
+                      generate-env --format env \
+                      --output /cachi2/cachi2.env /cachi2/output
+          echo "::endgroup::"
+        else
+          echo "No output directory found, skipping generate-env"
+          exit 1
+        fi
+
+        if [ -d ${{ env.LOCAL_CACHE_DIR }}/output ]; then
+          echo "::group::Injecting files"
+          podman run --rm -v "$PWD:/source:z" -v "$LOCAL_CACHE_DIR:/cachi2:z" -w /source "$HERMETO_IMAGE" \
+            --log-level DEBUG \
+            inject-files /cachi2/output || echo "Inject-files failed"
+          echo "::endgroup::"
Relevance

●● Moderate

No clear prior pattern on accepting/rejecting masking failures with || echo in CI steps.

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The action explicitly appends || echo ... to Hermeto commands, which neutralizes failures even
under set -e, so the workflow can continue after dependency caching errors as long as the output
directory exists.

.github/actions/docker-build/action.yaml[70-105]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The dependency-caching step masks Hermeto failures by converting non-zero exits into success via `|| echo ...`. This can allow stale or incomplete cache artifacts to be used later, producing unreliable build results.

## Issue Context
The step runs with `set -e`, but `cmd || echo ...` forces a zero exit status. The only guard is the existence of the output directory, not correctness/completeness of the generated env/deps.

## Fix Focus Areas
- .github/actions/docker-build/action.yaml[70-111]

## What to change
- Remove `|| echo "Fetch-deps failed"` and `|| echo "Inject-files failed"` so failures fail the workflow.
- Optionally, before fetching deps, delete/clean `${LOCAL_CACHE_DIR}/output` to avoid reusing stale output from previous attempts.
- Consider validating required artifacts exist (e.g., `/cachi2/output/deps/...` and `/cachi2/cachi2.env`) before proceeding to the build step.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools



Informational

5. Stale requirements path comments ✓ Resolved 🐞 Bug ⚙ Maintainability
Description
requirements-build.txt still contains autogenerated “via” comments referencing the old
.rhdh/docker/requirements-build.in path, which is misleading now that requirements files live at
the repo root. This won’t break builds but will confuse future updates/regeneration.
Code

requirements-build.txt[R1-21]

# This file was autogenerated by uv via the following command:
-#    uv pip compile --python-version=3.9 --python-platform=linux --generate-hashes --output-file=.rhdh/docker/requirements-build.txt .rhdh/docker/requirements-build.in
+#    uv pip compile --python-version=3.9 --python-platform=linux --generate-hashes --output-file=requirements-build.txt requirements-build.in
calver==2025.10.20 \
    --hash=sha256:c98b376c2424642224d456b2f70c51402343e008c63d204634665e1a2a2835f5 \
    --hash=sha256:d7d75224eed9d9263f4fb30008487615196d208d14752bfd93fea7af2c84f508
-    # via -r .rhdh/docker/requirements-build.in
+    # via -r requirements-build.in
editables==0.6 \
    --hash=sha256:1163834902381c4613787951c5914800fdf155ae08848a373b8ea5006780977c \
    --hash=sha256:d70e4698078a1d033e7786d9c64e5be070d058a67c21417024d38a58ac20aa43
    # via hatchling
flit-core==3.12.0 \
    --hash=sha256:18f63100d6f94385c6ed57a72073443e1a71a4acb4339491615d0f16d6ff01b2 \
    --hash=sha256:e7a0304069ea895172e3c7bb703292e992c5d1555dd1233ab7b5621b5b69e62c
-    # via -r .rhdh/docker/requirements-build.in
+    # via -r requirements-build.in
hatch-vcs==0.4.0 \
    --hash=sha256:093810748fe01db0d451fabcf2c1ac2688caefd232d4ede967090b1c1b07d9f7 \
    --hash=sha256:b8a2b6bee54cf6f9fc93762db73890017ae59c9081d1038a41f16235ceaf8b2c
-    # via -r .rhdh/docker/requirements-build.in
+    # via -r requirements-build.in
hatchling==1.21.1 \
    --hash=sha256:21e8c13f8458b219a91cb84e5b61c15bf786695d1c4fabc29e91e78f94bfe892 \
    --hash=sha256:bba440453a224e7d4478457fa2e8d8c3633765bafa02975a6b53b9bf917980bc
Relevance

●●● Strong

Repo often accepts small maintainability/typo fixes and requirements tooling cleanups.

PR-#57
PR-#144

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The file header indicates regeneration from requirements-build.in, but multiple # via lines
still reference .rhdh/docker/requirements-build.in, which no longer matches the repo layout.

requirements-build.txt[1-6]
requirements-build.txt[19-56]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Some autogenerated `# via -r ...` comments still reference the removed `.rhdh/docker/...` path.

## Issue Context
The file header was updated to root paths, but not all embedded comments were updated.

## Fix Focus Areas
- requirements-build.txt[19-56]

## What to change
- Regenerate `requirements-build.txt` using the documented command in `requirements-build.in`, or manually update the stale comment lines to `requirements-build.in` so the file is internally consistent.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

Context
⚠️ Tickets: not configured — ticket URL found in PR but could not be fetched — check ticket provider credentials
✅ Compliance rules (platform): 7 rules
✅ Cross-repo context
  Not relevant to this PR: redhat-developer/rhdh-chart

Grey Divider

Tip of the day
💡 Did you know, you can describe a rule in plain language on the Rules page and Qodo drafts it for you

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Grey Divider

Previous review results

Review updated until commit 59a5e2d ⚖️ Balanced

Results up to commit 76b298a


🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0) 🎨 UX issues (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)


Action required
1. Version arg not passed ✓ Resolved 🐞 Bug ≡ Correctness
Description
The hermetic build action builds the Containerfile without passing `--build-arg
RHDH_MUST_GATHER_VERSION=..., so images bake the default 0.0.0-unknown` into LABEL/ENV and
collection-scripts/version will report the wrong version. This is a regression from the previous
Makefile-driven build that set the build arg explicitly.
Code

.github/actions/docker-build/action.yaml[R152-164]

+    - name: "Build Docker Image"
+      id: build
+      uses: redhat-actions/buildah-build@7a95fa7ee0f02d552a32753e7414641a04307056 # v2.13
+      with:
+        containerfiles: ${{ steps.transform-containerfile.outputs.transformed_containerfile }}
+        context: .
+        platform: ${{ inputs.platform }}
+        tags: ${{ steps.meta.outputs.tags }}
+        labels: ${{ steps.meta.outputs.labels }}
+        extra-args: |
+          --network=none
+          --volume ${{ env.LOCAL_CACHE_DIR_REALPATH }}:/cachi2:z
+
Relevance

●● Moderate

Mixed history: PRs used version build-arg, but reviewer rejected changing default version ARG/ENV
semantics.

PR-#109
PR-#137

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The Containerfile’s version is derived from RHDH_MUST_GATHER_VERSION, but the new hermetic build
invocations never set the build arg, so the default 0.0.0-unknown becomes the baked image version
and is what the version script prints.

Containerfile[19-27]
Containerfile[72-73]
collection-scripts/version[5-10]
.github/actions/docker-build/action.yaml[152-164]
scripts/local-hermeto-build.sh[200-209]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The new hermetic build path never sets `RHDH_MUST_GATHER_VERSION` at build time. As a result, the image LABEL/ENV remains the Containerfile default (`0.0.0-unknown`) and `collection-scripts/version` will always print that incorrect value.

## Issue Context
`Containerfile` uses `ARG RHDH_MUST_GATHER_VERSION` to populate both LABEL and ENV. The composite build action uses `buildah-build` but does not provide any build args, and the local hermeto script similarly calls `podman build` without `--build-arg`.

## Fix Focus Areas
- .github/actions/docker-build/action.yaml[152-164]
- .github/workflows/container-build.yaml[87-96]
- .github/workflows/pr-build.yaml[79-88]
- scripts/local-hermeto-build.sh[200-209]
- Makefile[195-199]

## What to change
- Extend the composite action to accept a version/build-arg input (e.g. `mustGatherVersion`) and pass it through to the build command (either via a supported `build-args:` input on `redhat-actions/buildah-build` or by adding `--build-arg RHDH_MUST_GATHER_VERSION=...` to `extra-args`).
- Update `container-build.yaml` and `pr-build.yaml` to pass the computed version (e.g. `${{ steps.prep.outputs.VERSION }}`) into that new input.
- Update `scripts/local-hermeto-build.sh` (and the `Makefile` target) to pass `--build-arg RHDH_MUST_GATHER_VERSION=...` so `make image-build` behaves consistently.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools



Remediation recommended
2. Floating Hermeto image tag ✓ Resolved 🐞 Bug ⛨ Security
Description
Both the CI composite action and the local hermeto script use quay.io/konflux-ci/hermeto:latest,
so the supposedly hermetic pipeline is not reproducible and can change behavior or break without any
change in this repo. This also increases supply-chain risk because the build tool image is not
pinned to an immutable digest.
Code

.github/actions/docker-build/action.yaml[R64-69]

+    - name: Set up hermetic build variables
+      shell: bash
+      run: |
+        echo "HERMETO_IMAGE=quay.io/konflux-ci/hermeto:latest" >> $GITHUB_ENV
+        echo "LOCAL_CACHE_DIR=./hermeto-cache/rhdh-must-gather" >> $GITHUB_ENV
+
Relevance

●●● Strong

Team previously accepted removing floating :latest tags for reproducibility (pin UBI tags/digests).

PR-#137
PR-#181

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The CI action sets HERMETO_IMAGE to a :latest tag, and the local build script hardcodes the same
tag, making the toolchain mutable across time.

.github/actions/docker-build/action.yaml[64-69]
scripts/local-hermeto-build.sh[25-27]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Using the floating `:latest` tag for Hermeto makes builds non-deterministic and subject to upstream drift.

## Issue Context
Hermeto is the core dependency-fetch and injection tool for this pipeline; changes in the tool image affect the produced build inputs.

## Fix Focus Areas
- .github/actions/docker-build/action.yaml[64-69]
- scripts/local-hermeto-build.sh[25-27]

## What to change
- Replace `quay.io/konflux-ci/hermeto:latest` with an immutable reference (preferably `@sha256:<digest>`).
- Consider defining the pinned image reference once (e.g., repo-level env or shared script constant) and reusing it in both CI and `scripts/local-hermeto-build.sh`.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


3. Unpinned lockfile tool source ✓ Resolved 🐞 Bug ⛨ Security
Description
The new RPM lockfile workflow installs rpm-lockfile-prototype from refs/heads/main.zip, which
executes unpinned upstream code and can change behavior at any time. This makes automated lockfile
updates non-reproducible and increases supply-chain risk for a workflow that runs on schedule.
Code

.github/workflows/update-rpm-lockfile.yaml[R39-55]

+      - name: Install rpm-lockfile-prototype
+        run: |
+          if [[ ! -x "${HOME}/.local/bin/rpm-lockfile-prototype" ]]; then
+            echo "Installing rpm-lockfile-prototype ..."
+
+            sudo apt-get update
+            sudo apt-get install -y python3 python3-pip python3-dev build-essential
+            sudo apt-get install -y podman skopeo rpm
+            sudo apt-get install -y dnf python3-dnf
+
+            mkdir -p "${HOME}/.local/bin/"
+            python3 -m pip install --user https://github.com/konflux-ci/rpm-lockfile-prototype/archive/refs/heads/main.zip
+
+            # Update PATH
+            export PATH=${PATH%":${HOME}/.local/bin"}:${HOME}/.local/bin
+            echo "${HOME}/.local/bin" >> $GITHUB_PATH
+          else
Relevance

●●● Strong

Repo strongly prefers immutable/pinned refs in CI for supply-chain hygiene (many digest-pinned
action updates).

PR-#10
PR-#181
PR-#143

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The workflow explicitly installs from refs/heads/main.zip, which is not stable across runs and can
change without changes to this repository.

.github/workflows/update-rpm-lockfile.yaml[39-55]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The workflow installs `rpm-lockfile-prototype` from the upstream `main` branch ZIP, which is a moving target.

## Issue Context
This job runs on a schedule and can open PRs automatically; changes in the upstream `main` branch can silently change generated output or break the workflow.

## Fix Focus Areas
- .github/workflows/update-rpm-lockfile.yaml[39-55]

## What to change
- Pin the installation to an immutable reference:
 - prefer a tagged release if available, or
 - pin to a specific commit SHA archive URL, or
 - vendor/package the tool in a controlled way.
- Optionally, log the installed tool version/commit in the workflow summary for traceability.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


4. Hermeto failures partially masked ✓ Resolved 🐞 Bug ☼ Reliability
Description
The composite action suppresses errors from hermeto fetch-deps and hermeto inject-files using
|| echo, so the step may proceed even when Hermeto partially fails and leaves stale/invalid output
in the cache directory. This can create flaky or non-hermetic builds and makes the job logs
misleading (success path despite failed dependency caching).
Code

.github/actions/docker-build/action.yaml[R70-105]

+    - name: Cache dependencies with hermeto
+      shell: bash
+      run: |
+        set -ex
+
+        echo "::group::Creating local cache directory"
+        mkdir -p ${{ env.LOCAL_CACHE_DIR }} || echo "Failed to create local cache directory"
+        echo "::endgroup::"
+
+        echo "::group::Fetching dependencies with hermeto"
+        podman run --rm -v "$PWD:/source:z" -v "$LOCAL_CACHE_DIR:/cachi2:z" -w /source "$HERMETO_IMAGE" \
+          --log-level DEBUG \
+          fetch-deps --dev-package-managers \
+          --source . \
+          --output /cachi2/output \
+          '[{"type": "rpm", "path": "."}, {"type": "pip", "path": ".", "allow_binary": "false"}, {"type": "cargo", "path": "vendor/websocat"}]' || echo "Fetch-deps failed"
+        echo "::endgroup::"
+
+        if [ -d ${{ env.LOCAL_CACHE_DIR }}/output ]; then
+          echo "::group::Generating environment file"
+          podman run --rm -v "$PWD:/source:z" -v "$LOCAL_CACHE_DIR:/cachi2:z" -w /source "$HERMETO_IMAGE" \
+                      --log-level DEBUG \
+                      generate-env --format env \
+                      --output /cachi2/cachi2.env /cachi2/output
+          echo "::endgroup::"
+        else
+          echo "No output directory found, skipping generate-env"
+          exit 1
+        fi
+
+        if [ -d ${{ env.LOCAL_CACHE_DIR }}/output ]; then
+          echo "::group::Injecting files"
+          podman run --rm -v "$PWD:/source:z" -v "$LOCAL_CACHE_DIR:/cachi2:z" -w /source "$HERMETO_IMAGE" \
+            --log-level DEBUG \
+            inject-files /cachi2/output || echo "Inject-files failed"
+          echo "::endgroup::"
Relevance

●● Moderate

No clear prior pattern on accepting/rejecting masking failures with || echo in CI steps.

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The action explicitly appends || echo ... to Hermeto commands, which neutralizes failures even
under set -e, so the workflow can continue after dependency caching errors as long as the output
directory exists.

.github/actions/docker-build/action.yaml[70-105]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The dependency-caching step masks Hermeto failures by converting non-zero exits into success via `|| echo ...`. This can allow stale or incomplete cache artifacts to be used later, producing unreliable build results.

## Issue Context
The step runs with `set -e`, but `cmd || echo ...` forces a zero exit status. The only guard is the existence of the output directory, not correctness/completeness of the generated env/deps.

## Fix Focus Areas
- .github/actions/docker-build/action.yaml[70-111]

## What to change
- Remove `|| echo "Fetch-deps failed"` and `|| echo "Inject-files failed"` so failures fail the workflow.
- Optionally, before fetching deps, delete/clean `${LOCAL_CACHE_DIR}/output` to avoid reusing stale output from previous attempts.
- Consider validating required artifacts exist (e.g., `/cachi2/output/deps/...` and `/cachi2/cachi2.env`) before proceeding to the build step.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools



Informational
5. Stale requirements path comments ✓ Resolved 🐞 Bug ⚙ Maintainability
Description
requirements-build.txt still contains autogenerated “via” comments referencing the old
.rhdh/docker/requirements-build.in path, which is misleading now that requirements files live at
the repo root. This won’t break builds but will confuse future updates/regeneration.
Code

requirements-build.txt[R1-21]

# This file was autogenerated by uv via the following command:
-#    uv pip compile --python-version=3.9 --python-platform=linux --generate-hashes --output-file=.rhdh/docker/requirements-build.txt .rhdh/docker/requirements-build.in
+#    uv pip compile --python-version=3.9 --python-platform=linux --generate-hashes --output-file=requirements-build.txt requirements-build.in
calver==2025.10.20 \
    --hash=sha256:c98b376c2424642224d456b2f70c51402343e008c63d204634665e1a2a2835f5 \
    --hash=sha256:d7d75224eed9d9263f4fb30008487615196d208d14752bfd93fea7af2c84f508
-    # via -r .rhdh/docker/requirements-build.in
+    # via -r requirements-build.in
editables==0.6 \
    --hash=sha256:1163834902381c4613787951c5914800fdf155ae08848a373b8ea5006780977c \
    --hash=sha256:d70e4698078a1d033e7786d9c64e5be070d058a67c21417024d38a58ac20aa43
    # via hatchling
flit-core==3.12.0 \
    --hash=sha256:18f63100d6f94385c6ed57a72073443e1a71a4acb4339491615d0f16d6ff01b2 \
    --hash=sha256:e7a0304069ea895172e3c7bb703292e992c5d1555dd1233ab7b5621b5b69e62c
-    # via -r .rhdh/docker/requirements-build.in
+    # via -r requirements-build.in
hatch-vcs==0.4.0 \
    --hash=sha256:093810748fe01db0d451fabcf2c1ac2688caefd232d4ede967090b1c1b07d9f7 \
    --hash=sha256:b8a2b6bee54cf6f9fc93762db73890017ae59c9081d1038a41f16235ceaf8b2c
-    # via -r .rhdh/docker/requirements-build.in
+    # via -r requirements-build.in
hatchling==1.21.1 \
    --hash=sha256:21e8c13f8458b219a91cb84e5b61c15bf786695d1c4fabc29e91e78f94bfe892 \
    --hash=sha256:bba440453a224e7d4478457fa2e8d8c3633765bafa02975a6b53b9bf917980bc
Relevance

●●● Strong

Repo often accepts small maintainability/typo fixes and requirements tooling cleanups.

PR-#57
PR-#144

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The file header indicates regeneration from requirements-build.in, but multiple # via lines
still reference .rhdh/docker/requirements-build.in, which no longer matches the repo layout.

requirements-build.txt[1-6]
requirements-build.txt[19-56]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Some autogenerated `# via -r ...` comments still reference the removed `.rhdh/docker/...` path.

## Issue Context
The file header was updated to root paths, but not all embedded comments were updated.

## Fix Focus Areas
- requirements-build.txt[19-56]

## What to change
- Regenerate `requirements-build.txt` using the documented command in `requirements-build.in`, or manually update the stale comment lines to `requirements-build.in` so the file is internally consistent.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Results up to commit d9b130d ⚖️ Balanced


No changes from previous review

Qodo Logo

Add a reusable composite action for building container images
hermetically (--network none) with pre-fetched dependencies via
hermeto. This action handles disk space cleanup, dependency
prefetching, Containerfile transformation, and image building
with podman/buildah.

Assisted-by: Claude
The docker-build composite action had five issues preventing it from
working: missing generic fetcher for helm CGW binaries, a stale curl
step that conflicts with --network=none, a Containerfile transform
that would double-inject cachi2 patterns into the already-hermetic
.rhdh/docker/Containerfile, no requirements copy step for pip fetcher
discovery, and no workflow integration.

Fix all five issues in the composite action and add a hermetic-build
validation job to pr-build.yaml. This runs alongside the existing
non-hermetic build so Renovate PRs updating .rhdh/docker/requirements,
base image digests, or RPM lockfiles are validated before merge.

Locally verified: hermeto fetch-deps succeeds with all four fetcher
types (rpm, pip, generic, cargo) and helm binaries land at the path
install-helm-binary.sh --prefetch expects.

Assisted-by: Claude
@Fortune-Ndlovu
Fortune-Ndlovu force-pushed the RHIDP-14412-build-must-gather-image-hermetically-in-upstream-repo branch from d9b130d to 4a82cf8 Compare August 14, 2026 15:37
The previous lockfile had a stale openshift-clients aarch64 RPM URL
(build 202608051112) that returns 404 from the OCP mirror. Regenerate
to pick up the current build (202608132049) which resolves the hermetic
build failure in the fetch-deps RPM prefetch step.

Assisted-by: Claude
@Fortune-Ndlovu
Fortune-Ndlovu requested a review from rm3l August 14, 2026 15:54
@github-actions

Copy link
Copy Markdown
Contributor

PR images are available (for 1 week):

  1. quay.io/rhdh-community/rhdh-must-gather:pr-218
  2. quay.io/rhdh-community/rhdh-must-gather:pr-218-40a3dd0da

Convert both the publish (container-build.yaml) and PR
(pr-build.yaml) workflows to build images hermetically using the
docker-build composite action with --network=none. This ensures
upstream CI parity with downstream Konflux builds and catches
dependency issues early.

container-build.yaml: replace make image-push with the hermetic
composite action, build all tags in a single buildah invocation,
push individually, capture digest via podman push --digestfile.

pr-build.yaml: replace make image-build with the hermetic composite
action, remove the now-redundant hermetic-build validation job,
merge its file-change triggers into the main pr-build job.

Signed-off-by: Fortune Ndlovu <fndlovu@redhat.com>
Assisted-by: Claude
@Fortune-Ndlovu

Copy link
Copy Markdown
Member Author

/agentic_review

@rhdh-qodo-merge

Copy link
Copy Markdown

Code review by qodo was updated up to the latest commit 59a5e2d

@github-actions

Copy link
Copy Markdown
Contributor

PR images are available (for 1 week):

  1. quay.io/rhdh-community/rhdh-must-gather:pr-218
  2. quay.io/rhdh-community/rhdh-must-gather:pr-218-59a5e2d89

Comment thread .github/workflows/container-build.yaml Outdated
Comment thread .github/workflows/container-build.yaml Outdated
For consistency with all other RHDH repos, the main branch image
should be tagged "next" rather than "latest". The "latest" tag is
reserved for the current stable release.

Also update the expiry-tag detection to include next-* commit-SHA
tags so they still receive their 2-week expiration.

Assisted-by: Claude
Pin actions/checkout to SHA in container-build.yaml for supply-chain
safety, matching pr-build.yaml. Bump upload-artifact in the composite
action from v4.6.2 to v7 for consistency. Fix unquoted GITHUB_ENV
assignment in docker-build action. Map all step outputs through env:
blocks in container-build.yaml run: scripts to prevent potential
script injection, matching the pattern already used in pr-build.yaml.

Signed-off-by: Fortune Ndlovu <fndlovu@redhat.com>
@github-actions

Copy link
Copy Markdown
Contributor

PR images are available (for 1 week):

  1. quay.io/rhdh-community/rhdh-must-gather:pr-218
  2. quay.io/rhdh-community/rhdh-must-gather:pr-218-ccf8bb8a5

@github-actions

Copy link
Copy Markdown
Contributor

PR images are available (for 1 week):

  1. quay.io/rhdh-community/rhdh-must-gather:pr-218
  2. quay.io/rhdh-community/rhdh-must-gather:pr-218-7a06e0bb6

@github-actions

Copy link
Copy Markdown
Contributor

PR images are available (for 1 week):

  1. quay.io/rhdh-community/rhdh-must-gather:pr-218
  2. quay.io/rhdh-community/rhdh-must-gather:pr-218-40829c813

containerfilePath:
description: Path to the Containerfile to use
required: false
default: '.rhdh/docker/Containerfile'

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so we're still using two containefiles in https://github.com/redhat-developer/rhdh-must-gather/blob/main/Containerfile and https://github.com/redhat-developer/rhdh-must-gather/blob/main/.rhdh/docker/Containerfile ? I thought the idea here was to remove the need for two different files...

@nickboldt

Copy link
Copy Markdown
Member

have you prepared the downstream changes that will be needed to absorb this, once we get rid of the duplicate Containerfile in both the root and in .rhdh/docker/ ? (Aside: new convention is a folder called "containerfiles", rather than docker.

@Fortune-Ndlovu

Copy link
Copy Markdown
Member Author

/hold

@openshift-ci openshift-ci Bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. Used by Prow. label Aug 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. Used by Prow.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants