refactor(build): install Helm from CGW mirror instead of vendoring source

[nickboldt]

Summary

Build (Helm CGW binary)

• Remove vendor/helm/ source tree (~50k lines) and the go-toolset compile stage
• Install Helm v4.2.3 from mirror.openshift.com/pub/cgw/helm/ via artifacts.lock.yaml + Hermeto generic fetcher (Konflux) or curl by TARGETARCH (upstream Containerfile)
• Add hack/update-helm-lockfile.sh, hack/check-helm-binary-available.sh, hack/install-helm-binary.sh, and make helm-lockfile-update for version bumps
• CGW helm-builder as Containerfile Stage 2a; commented Stage 2b retains vendored go-toolset fallback when CGW lacks a release
• Fix CGW flat tarball extraction (helm-linux-amd64 at archive root, not linux-amd64/helm)
• Fix Makefile OS/ARCH ordering before HELM_ARCHIVE_DIR (broken E2E local-setup path on CI)

Supersedes the source-vendoring approach in draft PR #282.

Downstream

• Konflux on-pull PLR green via rhidp/rhdh !697

E2E fixes (Helm 4 + RHDH 2.x chart on Kind)

GitHub E2E failed after the Helm 4 upgrade for reasons unrelated to the CGW lockfile itself. Fixes on this branch:

• Resolve RHDH chart from oci://quay.io/rhdh/chart on all branches (auto-pick latest X.Y-N-CI via sort -V; drop flaky GitHub Releases tarball fetch)
• Branch Helm values on resolved chart major (1.x vs 2.x), not git branch name
• Wait for install-dynamic-plugins init to finish, then check backstage-backend for expected CreateContainerConfigError (not containerStatuses[0] during PodInitializing)
• Standalone deploy: quay.io/fedora/postgresql-15 on Kind (no registry.redhat.io pull secret)
• Helm 4 OCI pull status (Pulled: / Digest: on stdout): add helm_template_yaml() so helm template | kubectl apply validates
• Unit tests for chart version sort helpers (tests/e2e-chart-version.bats)

Related

• Bump workflow skill: rhdh-skill #86 (rhdh-must-gather-helm-bump)

Test plan

• Konflux hermetic build succeeds with generic prefetch (downstream !697)
• make test passes (unit + ShellCheck)
• GitHub Actions: Build PR Container Image
• GitHub Actions: E2E Tests (local mode)
• GitHub Actions: E2E Tests (container image)
• make image-build locally (optional; CI image job covers this)

Ref: https://redhat.atlassian.net/browse/RHIDP-16046

Generated-by: cursor

[nickboldt]

passed downstream in https://gitlab.cee.redhat.com/rhidp/rhdh/-/merge_requests/697 -> https://konflux-ui.apps.stone-prod-p02.hjvn.p1.openshiftapps.com/ns/rhdh-tenant/applications/rhdh-2/pipelineruns/rhdh-must-gather-2-on-pull-wlhhv

now just need to make the tests cooperate

[github-actions]

PR images are available (for 1 week):

1. quay.io/rhdh-community/rhdh-must-gather:pr-284
2. quay.io/rhdh-community/rhdh-must-gather:pr-284-bb2e0e811

[github-actions]

PR images are available (for 1 week):

1. quay.io/rhdh-community/rhdh-must-gather:pr-284
2. quay.io/rhdh-community/rhdh-must-gather:pr-284-bbd0fa414

[Fortune-Ndlovu]

/agentic_review

[rhdh-qodo-merge]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (1) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

1. Helm download unverified ✓ Resolved 🐞 Bug ⛨ Security

Description

Helm tarballs are downloaded and extracted in hack/install-helm-binary.sh (curl mode) and
hack/install-helm-local.sh without verifying the expected sha256, so a tampered archive could
install a malicious helm binary. This bypasses the checksum material already tracked in
artifacts.lock.yaml/sha256sum.txt.

Code

hack/install-helm-binary.sh[R68-71]

+    curl -fsSL "https://mirror.openshift.com/pub/cgw/helm/${HELM_VERSION}/${TARBALL}" \
+        -o "/tmp/${TARBALL}"
+    extract_helm "/tmp/${TARBALL}"
+    rm -f "/tmp/${TARBALL}"

Relevance

●●● Strong

Supply-chain hardening; repo already tracks Helm sha256s via lockfiles, so checksum verification
likely welcomed.

PR-#137

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The installer scripts perform curl + tar extraction with no checksum check, despite the repo
adding a lockfile containing sha256s and a helper that fetches sha256sum.txt.

hack/install-helm-binary.sh[50-72]
hack/install-helm-local.sh[35-44]
artifacts.lock.yaml[4-10]
hack/update-helm-lockfile.sh[37-48]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`hack/install-helm-binary.sh` and `hack/install-helm-local.sh` download `helm-<os>-<arch>.tar.gz` and extract it without validating the archive digest. The repo already pins digests for Linux artifacts in `artifacts.lock.yaml` and the lockfile updater reads `sha256sum.txt`, but the installer paths never check them.

### Issue Context
- Konflux prefetch path relies on `artifacts.lock.yaml` checksums, but curl/local paths currently do not.
- Prefer verifying against a *pinned* checksum (lockfile) where possible (linux/amd64, linux/arm64), rather than downloading a checksum from the same origin.

### Fix Focus Areas
- hack/install-helm-binary.sh[50-75]
- hack/install-helm-local.sh[35-64]
- artifacts.lock.yaml[1-10]
- Containerfile[15-27]

### Implementation notes
- Add checksum verification before `extract_helm`/`tar xzf`.
- For container curl-mode installs: copy `artifacts.lock.yaml` into the helm-builder stage and have the script read the expected checksum for `${TARBALL}` from it; then run `sha256sum -c` on the downloaded file.
- For local installs: if `${BUILD_OS}-${BUILD_ARCH}` is present in `artifacts.lock.yaml`, verify against it; otherwise either (a) fetch `sha256sum.txt` and verify, or (b) require an explicit expected checksum input for non-locked platforms.
- Ensure `sha256sum` is available in the image (install `coreutils` if needed on UBI minimal).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Missing tests/test_helper.bash source 📘 Rule violation ▣ Testability

Description

tests/e2e-chart-version.bats uses load 'test_helper' and does not source
tests/test_helper.bash as required, which can break shared test setup/fixtures. This violates the
requirement that all BATS tests source the common helper explicitly.

Code

tests/e2e-chart-version.bats[4]

+load 'test_helper'

Relevance

●●● Strong

Simple, deterministic compliance fix; teams typically accept standardizing test scaffolding/headers.

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

PR Compliance ID 1826 requires every .bats file to include a non-commented source/. statement
that references tests/test_helper.bash. In tests/e2e-chart-version.bats, line 4 uses `load
'test_helper' and there is no source tests/test_helper.bash` statement in the file header.

Rule 1826: BATS tests must source common test_helper.bash
tests/e2e-chart-version.bats[1-5]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The new BATS test `tests/e2e-chart-version.bats` does not source `tests/test_helper.bash` (it currently uses `load 'test_helper'`). The compliance rule requires an explicit `source`/`.` statement that includes `tests/test_helper.bash`.

## Issue Context
Repository contains the shared helper at `tests/test_helper.bash`; this file should be sourced near the top of every `.bats` file.

## Fix Focus Areas
- tests/e2e-chart-version.bats[1-6]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

3. E2E poll aborts early ✓ Resolved 🐞 Bug ☼ Reliability

Description

wait_for_helm_misconfigured_backstage_pods() uses kubectl | wc / kubectl | jq pipelines inside
assignments without tolerating non-zero kubectl exits, but run-e2e-tests.sh runs with `set -euo
pipefail`. A transient API error (or brief not-found during rollout) will therefore terminate the
E2E run instead of continuing to poll.

Code

tests/e2e/lib/test-utils.sh[R402-405]

+    while [ "$SECONDS" -lt "$deadline" ]; do
+        local count
+        count=$(kubectl -n "$namespace" get pods -l "$selector" -o jsonpath='{.items[*].metadata.name}' 2>/dev/null | wc -w)
+        if [ "$count" -ge "$expected_count" ]; then

Relevance

●● Moderate

Reliability improvement but behavior change in E2E polling; no close precedent for tolerating
kubectl failures under pipefail.

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The helper’s kubectl pipelines are executed in a shell with errexit+pipefail enabled (set by
the main runner). Under these options, any non-zero kubectl status will exit the script at the
assignment site.

tests/e2e/run-e2e-tests.sh[33-40]
tests/e2e/lib/test-utils.sh[396-405]
tests/e2e/lib/test-utils.sh[426-438]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
The new polling helper runs under the caller's `set -euo pipefail` and uses pipelines inside command substitutions/assignments. With `pipefail`, any non-zero `kubectl` exit makes the whole assignment fail and triggers `errexit`, aborting E2E instead of retrying.

### Issue Context
`tests/e2e/run-e2e-tests.sh` enables `set -euo pipefail` before sourcing `tests/e2e/lib/test-utils.sh`, so functions inherit these options.

### Fix Focus Areas
- tests/e2e/run-e2e-tests.sh[33-40]
- tests/e2e/lib/test-utils.sh[402-449]

### Implementation notes
- Avoid `kubectl ... | wc -w` directly under pipefail. Instead:
 - `pods=$(kubectl ... 2>/dev/null || true); count=$(wc -w <<<"$pods")`
- For `kubectl ... | jq ...` queries, first capture pod JSON with `kubectl ... -o json 2>/dev/null || { all_ready=false; continue; }`, then run `jq` on the captured JSON (or add `|| true` and treat empty as not-ready).
- Ensure the polling loop continues on transient kubectl errors, but still fails fast on confirmed init pull errors (the existing describe+return path can remain).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

Context

⚠️ Tickets: not configured — ticket URL found in PR but could not be fetched — check ticket provider credentials

✅ Compliance rules (platform): 7 rules

✅ Cross-repo context

  Explored: repo: redhat-developer/rhdh-chart (sha: ea6b1aef)

Tip of the day

💡 Did you know, you can type 'qodo, fix this' on a finding and the fix lands right on your PR

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

[Fortune-Ndlovu]

Hey @nickboldt, just flagging the dependency chain across the PRs on this initiative (RHIDP-16046):

1. chore(build): add RPM lockfile workflow and vendor Helm from source [RHIDP-16046] #178 (@rm3l , merged Aug 6), vendored Helm from source, similar to how websocat is handled
2. refactor(build): install Helm from CGW mirror instead of vendoring source #284 (this PR), builds on 178 by replacing the vendored source tree with the lighter CGW binary approach
3. feat(ci): build must-gather image hermetically in upstream repo [RHIDP-14412] #218 (mine), adds a hermetic build pipeline with Hermeto, which touches overlapping Containerfile and build infrastructure

Your PR supersedes the Helm-related parts of both 178 and 218. Once this lands, I'll rebase my 218 to reconcile with your CGW binary approach instead of the COPY bin/helm method I had. No action needed on your side, just wanted to make sure we're aware of the overlap so we don't step on each other during merge.

[Fortune-Ndlovu]

thanks for pushing the PR @nickboldt added a few nits

[github-actions]

PR images are available (for 1 week):

1. quay.io/rhdh-community/rhdh-must-gather:pr-284
2. quay.io/rhdh-community/rhdh-must-gather:pr-284-8557526e2