feat(skills): add rhdh-must-gather-helm-bump skill and script

.claude-plugin/marketplace.json

@@ -44,6 +44,7 @@
         "./skills/ci/rhdh-prow-jobs",
         "./skills/ci/rhdh-prow-release-branch",
         "./skills/ci/rhdh-prow-trigger",
+        "./skills/ci/rhdh-must-gather-helm-bump",
         "./skills/ci/rhdh-yarn-bump"
       ]
     },

README.md

@@ -48,7 +48,7 @@ one. It performs no work itself.
 and the single source of truth for membership. This file does not restate it.
 
 Two skills are human-invoked and never selected automatically: `/ask-rhdh` and
-`/setup-rhdh-skills`. The other 39 are model-invoked, and can also be called by
+`/setup-rhdh-skills`. The other 40 are model-invoked, and can also be called by
 name.
 
 Skills are grouped into six folders:
@@ -57,7 +57,7 @@ Skills are grouped into six folders:
 | --- | --- |
 | `jira/` | Creating, refining, updating, and reporting on RHIDP, RHDHPLAN, RHDHBUGS, and RHDHSUPP work, plus sprint ceremonies and linking PRs to issues. |
 | `plugins/` | Authoring, wiring, exporting, and fixing Backstage dynamic plugins; the overlays repository; local RHDH; opening and reviewing pull requests; midstream propagation. |
-| `ci/` | Prow job configuration and nightly triggers, Konflux and Tekton task updates, base images, and Yarn bumps. |
+| `ci/` | Prow job configuration and nightly triggers, Konflux and Tekton task updates, base images, must-gather Helm bumps, and Yarn bumps. |
 | `release/` | Release status and readiness, milestone schedules, freeze announcements, teams, test-plan review, platform lifecycle, and the plugin CVE export. |
 | `reference/` | The reusable layer other skills invoke by name: repository and version context, the forge read seam, the write gate, and the Jira and Backstage reference material. |
 | `meta/` | The two human-invoked entry points, plus skill authoring and repository agent-readiness. |

skills/ci/rhdh-must-gather-helm-bump/SKILL.md

@@ -0,0 +1,108 @@
+---
+name: rhdh-must-gather-helm-bump
+description: >-
+  Bumps the Helm CLI baked into redhat-developer/rhdh-must-gather, mirrors
+  helm-related files into gitlab.cee.redhat.com/rhidp/rhdh distgit, patches
+  Konflux Tekton prefetch (CGW generic vs vendored gomod), and updates
+  sync/upstream_SHA. Use when bumping must-gather's HELM_VERSION (upstream or
+  downstream), switching between CGW binary and vendored Helm build paths, or
+  diagnosing a Konflux rhdh-must-gather prefetch failure after a Helm release.
+---
+
+# RHDH must-gather Helm bump
+
+## Goal
+
+Propagate a **Helm CLI** version bump across upstream GitHub and midstream GitLab:
+
+| Repo | Path | What changes |
+|------|------|--------------|
+| [`redhat-developer/rhdh-must-gather`](https://github.com/redhat-developer/rhdh-must-gather) | repo root | `Makefile` `HELM_VERSION`, `artifacts.lock.yaml` **or** `vendor/helm/`, Stage 2a/2b, hack scripts |
+| [`gitlab.cee.redhat.com/rhidp/rhdh`](https://gitlab.cee.redhat.com/rhidp/rhdh) | `distgit/containers/rhdh-must-gather/` | Mirror helm-related files; regenerate hermetic `Containerfile` from `.rhdh/docker/Containerfile` |
+| Same (midstream) | `.tekton/rhdh-must-gather-2-{pull,push}.yaml` | `prefetch-input` (`generic` vs `gomod`) |
+| Same | `.tekton-templates/components.yaml` | `must-gather.prefetch_input` only |
+| Same | `sync/upstream_SHA_rhdh-must-gather` | Upstream commit SHA after sync |
+
+## Essential principles
+
+- **Helm CLI ≠ RHDH chart version.** This skill only bumps the CLI in the must-gather image, not `oci://quay.io/rhdh/chart`.
+- **CGW binary path is preferred** when `mirror.openshift.com/pub/cgw/helm/<version>/` has linux amd64/arm64 tarballs — smaller tree, faster Konflux builds.
+- **Execute the bundled script** ([scripts/bump-must-gather-helm.py](scripts/bump-must-gather-helm.py)); do not reimplement lockfile, Stage flip, or Tekton edits inline.
+- **`--check` and `--dry-run` are the preview.** Apply is a write: follow `/mutation-gate`. Approval to preview is not approval to write.
+- **Commit / PR / MR only when the user asks.**
+
+## Prerequisites
+
+- Python 3.9+, `bash`, `curl`, `git`, `rsync`
+- Network access to CGW mirror and both repo checkouts
+- Clean git trees in upstream and downstream (or pass `--allow-dirty`)
+- Upstream checkout on a branch that contains `hack/update-helm-lockfile.sh`
+
+## Usage
+
+```bash
+SKILL=<this skill's directory>
+
+# Probe CGW + planned install path
+python3 "${SKILL}/scripts/bump-must-gather-helm.py" --to 4.3.0 --check --parent-dir ~/RHDH
+
+# Dry-run, then apply (apply only after /mutation-gate)
+python3 "${SKILL}/scripts/bump-must-gather-helm.py" --to 4.3.0 --dry-run --parent-dir ~/RHDH
+python3 "${SKILL}/scripts/bump-must-gather-helm.py" --to 4.3.0 --parent-dir ~/RHDH
+```
+
+Full flag reference: `python3 "${SKILL}/scripts/bump-must-gather-helm.py" --help`.
+
+**Default:** updates upstream, syncs helm-related paths into distgit, flips Stage 2a/2b, regenerates distgit `Containerfile`, patches Tekton prefetch, writes `sync/upstream_SHA_rhdh-must-gather`. Does **not** commit, push, or open PR/MR.
+
+## Workflow
+
+1. Confirm target `--to` version against [Helm releases](https://github.com/helm/helm/releases).
+2. Resolve `--upstream` / `--downstream` (or `--parent-dir`).
+3. Run `--check` and `--dry-run`. That pair is the preview.
+4. Apply only after `/mutation-gate` approval, with upstream and distgit/Tekton as separate operations. Approval to dry-run is not approval to write.
+5. Review `git diff` in both repos, then the gates in [references/verification.md](references/verification.md).
+6. Commit / PR or MR only when the user asks, again through `/mutation-gate`.
+
+## Completion
+
+`--check` and `--dry-run` are the preview. They are not a write and do not
+approve an apply.
+
+An apply is done when `/mutation-gate` approved the writes (upstream and
+distgit/Tekton as separate operations), the script ran without `--dry-run`,
+and the trees contain:
+
+- upstream `HELM_VERSION` plus lockfile or `vendor/helm/` matching `--to`
+- Stage 2a active on `mode=cgw`, Stage 2b on `mode=vendor` (upstream + distgit)
+- distgit helm-related files synced; `vendor/helm` absent on `mode=cgw`
+- Tekton `prefetch-input` matching install path (`generic` vs `gomod`) for
+  must-gather only (other `components.yaml` entries untouched)
+- `sync/upstream_SHA_rhdh-must-gather` pointing at a **committed** upstream HEAD
+  (the script refuses to pin a dirty HEAD)
+
+Report `mode=`, paths touched, and any verification still pending
+([references/verification.md](references/verification.md)). Working trees stay
+uncommitted; if the user asks for a commit, PR, or MR, follow `/mutation-gate`
+and attach Jira via `/rhdh-jira-link`.
+
+## Anti-patterns
+
+- Blaming a Helm bump when only GitHub E2E fails — triage chart/E2E harness first ([references/verification.md](references/verification.md)).
+
+## Reference index
+
+| Reference | Load when... |
+|-----------|--------------|
+| [references/install-paths.md](references/install-paths.md) | Choosing or switching CGW binary vs vendored source (Stage 2a/2b) |
+| [references/tekton-prefetch.md](references/tekton-prefetch.md) | Editing or debugging Konflux `prefetch-input` JSON |
+| [references/verification.md](references/verification.md) | Validating the bump (unit tests, Konflux, E2E triage) |
+| [references/helm4-notes.md](references/helm4-notes.md) | Debugging E2E after a major Helm version bump |
+
+## Related
+
+- Upstream `make vendor` / `make helm-lockfile-update` — same logic, upstream only
+- `/rhdh-base-images` — UBI/RPM bumps for overlapping repos
+- `/rhdh-konflux-tasks` — task-bundle digest bumps in the same midstream `.tekton` trees
+- CGW binary approach: [rhdh-must-gather#284](https://github.com/redhat-developer/rhdh-must-gather/pull/284)
+- Vendored fallback reference: [rhdh-must-gather#282](https://github.com/redhat-developer/rhdh-must-gather/pull/282)

skills/ci/rhdh-must-gather-helm-bump/agents/openai.yaml

@@ -0,0 +1,3 @@
+interface:
+  display_name: "RHDH Must-Gather Helm Bump"
+  short_description: "Bump Helm CLI in rhdh-must-gather upstream and midstream distgit/Tekton"

skills/ci/rhdh-must-gather-helm-bump/references/helm4-notes.md

@@ -0,0 +1,23 @@
+# Helm 4 notes for must-gather
+
+Helm 4 (currently installed from CGW in must-gather) differs from Helm 3 in ways that affect **E2E**, not the bump script itself.
+
+## OCI pull status on stdout
+
+When templating `oci://` charts, Helm 4 prints lines like:
+
+```text
+Pulled: quay.io/rhdh/chart:2.0-59-CI
+Digest: sha256:...
+---
+apiVersion: v1
+```
+
+Piping `helm template … | kubectl apply` fails validation (`apiVersion not set, kind not set`) unless those status lines are stripped. Upstream E2E uses `helm_template_yaml()` in `tests/e2e/lib/test-utils.sh`.
+
+Must-gather **collection** scripts do not pipe template output to kubectl today. If a future collector adds that pattern, filter pull lines or use `--output-dir` + `kubectl apply -f`.
+
+## When this matters for a bump
+
+- Major Helm upgrades (3 → 4): re-run upstream E2E; expect harness fixes separate from lockfile bumps.
+- Patch/minor bumps within Helm 4: unlikely to need E2E changes unless CGW tarball layout or CLI flags change.

skills/ci/rhdh-must-gather-helm-bump/references/install-paths.md

@@ -0,0 +1,35 @@
+# CGW binary vs vendored Helm source
+
+The bump script probes `hack/check-helm-binary-available.sh <version>` against `mirror.openshift.com/pub/cgw/helm/`.
+
+## Decision table
+
+| CGW mirror has linux amd64/arm64? | Upstream action | Konflux prefetch | Containerfile |
+|-----------------------------------|-----------------|------------------|---------------|
+| **Yes** (preferred) | `hack/update-helm-lockfile.sh` | `generic` on distgit root (`artifacts.lock.yaml`) | Stage **2a** active (CGW binary) |
+| **No** | `hack/update-vendor.sh helm` | `gomod` on `vendor/helm` | Stage **2b** active (go-toolset build) |
+
+The script flips Stage 2a/2b bidirectionally on:
+
+- upstream `Containerfile`
+- upstream `.rhdh/docker/Containerfile`
+- distgit `Containerfile` (after regenerating it from `.rhdh/docker/Containerfile`)
+
+## Distgit sync (what the script copies)
+
+- `Makefile`, `artifacts.lock.yaml`
+- entire `hack/`
+- `.rhdh/docker/Containerfile`
+- entire `vendor/` — on `mode=cgw`, **omit/delete** `vendor/helm` (other vendor trees such as `websocat` stay)
+
+It does **not** copy upstream root `Containerfile` onto distgit. Distgit root `Containerfile` is regenerated from `.rhdh/docker/Containerfile`, preserving existing `RHDH_MUST_GATHER_VERSION` and the Brew/Konflux metadata footer (`ENV SUMMARY=` / `LABEL` block) that `sync-midstream.sh` appends. On each regenerate it increments `release="N"` → `N+1` and updates `konflux.additional-tags` so the `version-release` suffix matches with a numeric boundary (`2.0-1` → `2.0-2`, and `2.0-10` → `2.0-11` not `2.0-20`).
+
+## CGW tarball layout
+
+CGW publishes flat tarballs: member `helm-linux-amd64` (or `helm-linux-arm64`) at the archive root, **not** `linux-amd64/helm`. Upstream `hack/install-helm-binary.sh` extracts the correct member; lockfile entries use filenames `helm-linux-amd64.tar.gz` and `helm-linux-arm64.tar.gz`.
+
+If Konflux prefetch install fails after a bump, verify tarball layout before blaming Hermeto.
+
+## Stage 2a / 2b (script-owned)
+
+When `mode=vendor`, Stage 2a lines are commented and Stage 2b lines are uncommented. When `mode=cgw`, the reverse. Doc-only comments (`# Comment this out…`, `# https://…`, `# update via…`) stay commented.

skills/ci/rhdh-must-gather-helm-bump/references/tekton-prefetch.md

@@ -0,0 +1,39 @@
+# Tekton prefetch for rhdh-must-gather
+
+Konflux `prefetch-dependencies` input is a JSON array on PipelineRun param `prefetch-input` and in `.tekton-templates/components.yaml` under `must-gather.prefetch_input` only (other components in that file must not change).
+
+## CGW binary (Stage 2a — default)
+
+Helm linux-amd64/arm64 tarballs are listed in `artifacts.lock.yaml`. Hermeto **generic** fetcher prefetches them.
+
+CGW tarballs use a flat layout (`helm-linux-amd64` at archive root). See [install-paths.md](install-paths.md) for Stage 2a/2b and distgit sync rules.
+
+```json
+[
+  {"type": "rpm", "path": "distgit/containers/rhdh-must-gather"},
+  {"type": "pip", "path": "distgit/containers/rhdh-must-gather", "allow_binary": "false"},
+  {"type": "generic", "path": "distgit/containers/rhdh-must-gather"},
+  {"type": "cargo", "path": "distgit/containers/rhdh-must-gather/vendor/websocat"}
+]
+```
+
+Files the script patches in `rhidp/rhdh`:
+
+- `.tekton/rhdh-must-gather-2-pull.yaml` — `spec.params` → `prefetch-input`
+- `.tekton/rhdh-must-gather-2-push.yaml` — same
+- `.tekton-templates/components.yaml` — `must-gather.prefetch_input` only
+
+## Vendored source (Stage 2b)
+
+When CGW has no binaries, helm is built from `vendor/helm` with go-toolset. Use **gomod** prefetch:
+
+```json
+[
+  {"type": "rpm", "path": "distgit/containers/rhdh-must-gather"},
+  {"type": "pip", "path": "distgit/containers/rhdh-must-gather", "allow_binary": "false"},
+  {"type": "gomod", "path": "distgit/containers/rhdh-must-gather/vendor/helm"},
+  {"type": "cargo", "path": "distgit/containers/rhdh-must-gather/vendor/websocat"}
+]
+```
+
+Stage 2a/2b Containerfile swap is owned by the bump script — see [install-paths.md](install-paths.md). Regenerating PLRs from templates is **not** required for prefetch-only edits — patch the three files above directly.

skills/ci/rhdh-must-gather-helm-bump/references/verification.md

@@ -0,0 +1,31 @@
+# Verification after a must-gather Helm bump
+
+## Upstream (GitHub)
+
+| Gate | Command / location | Pass when |
+|------|-------------------|-----------|
+| Unit tests | `make test` in `rhdh-must-gather` | BATS suite green |
+| Local helm | `make local-setup && ./bin/helm version` | Version matches `HELM_VERSION`; tarball checksum verified |
+| Image build | GitHub Actions “Build PR Container Image” | Image builds; `helm version` works in container |
+| E2E | GitHub Actions “E2E Tests” | Kind E2E completes |
+
+Konflux hermetic builds are validated downstream; upstream CI is the primary gate for script and E2E harness changes.
+
+## Downstream (GitLab midstream)
+
+| Gate | Command / location | Pass when |
+|------|-------------------|-----------|
+| Distgit sync | `git diff distgit/containers/rhdh-must-gather` | Helm files match upstream, including `hack/verify-helm-tarball.sh` |
+| Upstream SHA | `sync/upstream_SHA_rhdh-must-gather` | Matches upstream commit synced |
+| Konflux | `rhdh-must-gather-2-on-pull` PipelineRun | Prefetch + build succeed |
+
+## E2E triage (upstream only)
+
+If unit tests and image build pass but **E2E fails**, the Helm bump may be unrelated. Common false positives from [rhdh-must-gather#284](https://github.com/redhat-developer/rhdh-must-gather/pull/284):
+
+- Chart resolution from `oci://quay.io/rhdh/chart` (not GitHub Releases)
+- RHDH 2.x chart values and init-container pod lifecycle in E2E waits
+- Helm 4 printing `Pulled:` / `Digest:` to stdout on `helm template oci://…` (see [helm4-notes.md](helm4-notes.md))
+- `wait_for_*` loops under `set -o pipefail`: capture `kubectl` with `|| true` and avoid piping status JSON under pipefail
+
+Check E2E job logs for the first `ERROR` or `make: ***` line — ignore debug must-gather SUCCESS lines from the EXIT trap.