Skip to content

OAPE-906: add automated CVE triage tool for helm-operator #459

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mytreya-rh wants to merge 1 commit into
base: main
Choose a base branch
from
+2,790 −2
Open

OAPE-906: add automated CVE triage tool for helm-operator #459

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,9 @@
/dist
**/bin/

# CVE triage tool scratch output (hack/cve-triage)
/.work/

# Test artifacts
**/testbin/

Expand Down Expand Up @@ -128,5 +131,5 @@ tags

# End of https://www.toptal.com/developers/gitignore/api/go,vim,emacs,visualstudiocode

# Python cache (Ansible molecule)
test/ansible/plugins/filter/__pycache__/
# Python cache
**/__pycache__/
279 changes: 279 additions & 0 deletions hack/cve-triage/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,279 @@
# CVE Triage — Go Binary Reachability Analysis

Automated triage of CVE Jira tickets for Go operator images. Proves a binary
is **not affected** by a CVE by running symbol-level `govulncheck` and a VTA
call graph analysis, then closes the ticket as *Not a Bug* with full evidence.

---

## How it works

Scanner alerts fire on any Go package present in a binary, regardless of
whether the vulnerable code is actually called. This tool proves non-reachability
at two independent levels:

1. **govulncheck** — symbol-level scan: confirms the vulnerable symbols are not
in any reachable call chain originating from `main()` or `init()`.
2. **VTA callgraph** — static call graph: `digraph somepath <main> <symbol>`
confirms no execution path exists to any vulnerable symbol. A sanity check
against known-reachable nodes validates the result before trusting any "no path"
verdict.

If either check finds reachability, the tool exits with code 2 and does **not**
close the ticket.

---

## Files

| File | Description |
|------|-------------|
| `cve_triage_core.py` | Reusable library — all analysis logic, no product-specific values |
| `triage_helm_operator_cve.py` | Helm-operator driver — supplies component config, parses CLI |
| `README.md` | This file |

---

## Quick start

### Prerequisites

```bash
go install golang.org/x/vuln/cmd/govulncheck@latest
go install golang.org/x/tools/cmd/callgraph@latest
go install golang.org/x/tools/cmd/digraph@latest
```

### Credentials

Set once in your shell profile (or pass as flags each time):

```bash
export JIRA_EMAIL=you@redhat.com
export JIRA_TOKEN=<your-atlassian-api-token>
```

Alternatively, store email and token in `~/email` and `~/jira_token` (file
fallback). Resolution order: CLI flag → env var → file.

The Jira base URL (`https://redhat.atlassian.net`) is a hardcoded constant in
`cve_triage_core.py`. It is intentionally **not** configurable via a CLI flag
or environment variable — accepting an externally-controlled destination host
there would let the tool's Basic-auth credentials and issue data be sent to
an arbitrary server (SSRF). See "Security hardening" below.

### Run

```bash
# Always dry-run first — no Jira changes, full analysis printed to stdout.
# Works even on already-closed tickets.
python3 hack/cve-triage/triage_helm_operator_cve.py OCPBUGS-XXXXX --dry-run

# Apply (posts comment + closes ticket)
python3 hack/cve-triage/triage_helm_operator_cve.py OCPBUGS-XXXXX

# Also accepts full Jira URL
python3 hack/cve-triage/triage_helm_operator_cve.py \
"https://redhat.atlassian.net/browse/OCPBUGS-XXXXX"
```

---

## CLI reference

```
python3 hack/cve-triage/triage_helm_operator_cve.py <issue> [options]
```
Comment on lines +84 to +86

@coderabbitai coderabbitai Bot Aug 6, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Add languages to the fenced code blocks.

markdownlint reports MD040 for the CLI and workflow blocks. Mark the blocks as text or console.

Proposed fix 
-```
+```text
 python3 hack/cve-triage/triage_helm_operator_cve.py <issue> [options]

- +text
[1] Fetch Jira

</details>







Also applies to: 133-150

<details>
<summary>🧰 Tools</summary>

<details>
<summary>🪛 markdownlint-cli2 (0.23.2)</summary>

[warning] 84-84: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

</details>

</details>

<details>
<summary>🤖 Prompt for AI Agents</summary>

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @hack/cve-triage/README.md around lines 84 - 86, Add a language identifier
such as text or console to the fenced code blocks in the README, including the
CLI example and the workflow block around “[1] Fetch Jira” and lines 133–150, so
all blocks satisfy markdownlint MD040.


</details>

<!-- fingerprinting:phantom:triton:caracal -->

<!-- cr-indicator-types:potential_issue -->

<!-- cr-comment:v1:131043c25d5b9198b92807d9 -->

_Source: Linters/SAST tools_

<!-- This is an auto-generated comment by CodeRabbit -->


### Positional

| Argument | Description |
|----------|-------------|
| `issue` | OCPBUGS key (`OCPBUGS-12345`) or full Jira URL |

### Jira credentials

| Flag | Env var | File fallback | Description |
|------|---------|---------------|-------------|
| `--jira-email EMAIL` | `JIRA_EMAIL` | `~/email` | Analyst email |
| `--jira-token TOKEN` | `JIRA_TOKEN` | `~/jira_token` | Atlassian API token |

The Jira base URL is not configurable — see "Security hardening" below.

### Git options

| Flag | Default | Description |
|------|---------|-------------|
| `--remote NAME` | auto-detected | Git remote name for the upstream repo. Auto-detected by matching `openshift/ocp-release-operator-sdk` in `git remote -v`. |

### Analysis options

| Flag | Description |
|------|-------------|
| `--dry-run` | Print full analysis without making any Jira API calls. Skips the already-closed early exit so you can re-analyse a closed ticket. |
| `--branch BRANCH` | Deprecated/no-op — kept for CLI compatibility. The release branch is always derived from Jira `affectedVersion`; see "Security hardening" below. |
Comment on lines +113 to +114

@coderabbitai coderabbitai Bot Aug 6, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Describe --dry-run as read-only, not offline.

The workflow still fetches Jira data to obtain the CVE, affected version, component, and status. The CLI context also describes dry-run as avoiding Jira mutations, not all Jira API calls. Update this text to avoid implying that credentials or network access are unnecessary.

Proposed wording 
-| `--dry-run` | Print full analysis without making any Jira API calls. Skips the already-closed early exit so you can re-analyse a closed ticket. |
+| `--dry-run` | Print full analysis without posting or closing Jira issues. Skips the already-closed early exit so you can re-analyse a closed ticket. |
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
| `--dry-run` | Print full analysis without making any Jira API calls. Skips the already-closed early exit so you can re-analyse a closed ticket. |
| `--branch BRANCH` | Deprecated/no-op — kept for CLI compatibility. The release branch is always derived from Jira `affectedVersion`; see "Security hardening" below. |
| `--dry-run` | Print full analysis without posting or closing Jira issues. Skips the already-closed early exit so you can re-analyse a closed ticket. |
| `--branch BRANCH` | Deprecated/no-op — kept for CLI compatibility. The release branch is always derived from Jira `affectedVersion`; see "Security hardening" below. |
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@hack/cve-triage/README.md` around lines 113 - 114, Update the --dry-run entry
in the README to describe the mode as read-only and preventing Jira mutations,
while retaining that it prints the full analysis and permits re-analysis of
closed tickets. Remove the claim that it makes no Jira API calls, since the
workflow still reads Jira data and requires access.

| `--no-callgraph` | Skip the VTA callgraph step. Use when govulncheck alone is sufficient. |
| `--force` | Re-run govulncheck / callgraph even if cached output exists in `.work/`. |

`--worktree-base` is not exposed as a CLI flag — it is always derived from
the hardcoded `helm-operator` binary name (`/tmp/helm-operator-cve-worktrees`).

### Exit codes

| Code | Meaning |
|------|---------|
| `0` | Not Affected — Jira closed (or dry-run showed same) |
| `1` | Error or precondition failure |
| `2` | CVE IS reachable — fix required, Jira **not** closed |

---

## Workflow (8 steps)

```
[1] Fetch Jira CVE ID, affected version, PS component, status
[2] Verify Go CVE Jira heuristics + OSV / vuln.go.dev lookup
├─ Not a Go CVE → exit 1
├─ Stdlib CVE → find golang-builder-container ticket, mark duplicate
└─ No vuln DB entry yet → module-presence check (go list -deps)
[3] DB awareness govulncheck -version; warn if DB predates CVE modification date
[4] Git worktree git fetch <remote> release-4.XX
git worktree add --detach (isolated; current branch untouched)
[5] govulncheck govulncheck -json ./cmd/<binary>/
CVE found at symbol level → exit 2
[6] Callgraph VTA callgraph -algo vta -format=digraph ./cmd/<binary>/
Sanity-check digraph on known-reachable nodes
digraph somepath <main> <each vulnerable symbol>
Any path found → exit 2
[7] Write report .work/compliance/analyze-cve/<CVE>/<report>.md
[8] Close Jira Post evidence comment → transition to Closed / Not a Bug
```

---

## Artifacts

All output is written to `.work/compliance/analyze-cve/<CVE-ID>/`:

| File | Description |
|------|-------------|
| `govulncheck-4.XX.json` | Raw govulncheck JSON stream |
| `callgraph-4.XX.txt` | Raw VTA callgraph (digraph format) |
| `report-helm-operator-4.XX.md` | Structured analysis report |

Cached files are reused on subsequent runs (skip re-running the slow callgraph
step). Use `--force` to regenerate.

---

## Tips

- Run `--dry-run` on a **closed** ticket to audit the analysis that was
previously run, or to verify the tool would still reach the same verdict on
the current branch state.
- The callgraph step takes 2–5 minutes. Use `--no-callgraph` when the
govulncheck result alone is sufficient (govulncheck not found → binary is clean).
Comment on lines +174 to +175

@coderabbitai coderabbitai Bot Aug 6, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail
rg -n -C 8 'govulncheck|FileNotFoundError|CalledProcessError|not found|Not Affected|return 0' \
  hack/cve-triage/cve_triage_core.py

Repository: openshift/ocp-release-operator-sdk

Length of output: 34125

🏁 Script executed:

#!/bin/bash
set -euo pipefail
printf '%s\n' '--- README context ---'
cat -n hack/cve-triage/README.md | sed -n '150,190p'
printf '%s\n' '--- core definitions and call sites ---'
cat -n hack/cve-triage/cve_triage_core.py | sed -n '922,975p;1898,1930p;2098,2132p;2158,2170p;2240,2290p'
printf '%s\n' '--- related drivers, tests, and documentation ---'
rg -n -C 4 'no-callgraph|run_govulncheck|GOVULNCHECK|returncode|Not Affected|Not a Bug' hack/cve-triage

Repository: openshift/ocp-release-operator-sdk

Length of output: 31030

🏁 Script executed:

#!/bin/bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
p = Path("hack/cve-triage/cve_triage_core.py")
text = p.read_text()
for needle in ("def run_govulncheck", "if govulncheck_found:", "if not no_cg_this_ver:", "return 0"):
    print(f"{needle}: {text.count(needle)} occurrence(s)")
PY

Repository: openshift/ocp-release-operator-sdk

Length of output: 314

🏁 Script executed:

#!/bin/bash
set -euo pipefail
printf '%s\n' '--- driver entrypoint and exception handling ---'
cat -n hack/cve-triage/triage_helm_operator_cve.py | tail -n 70
printf '%s\n' '--- all invocations of run_triage and top-level handlers ---'
rg -n -C 8 'run_triage|except Exception|SystemExit|sys\.exit|return result' hack/cve-triage --glob '*.py'
printf '%s\n' '--- AST check of run_govulncheck subprocess handling ---'
python3 - <<'PY'
import ast
from pathlib import Path
source = Path("hack/cve-triage/cve_triage_core.py").read_text()
tree = ast.parse(source)
fn = next(n for n in tree.body if isinstance(n, ast.FunctionDef) and n.name == "run_govulncheck")
runs = [n for n in ast.walk(fn) if isinstance(n, ast.Call)
        and isinstance(n.func, ast.Attribute) and n.func.attr == "run"]
print("subprocess.run calls:", len(runs))
print("returncode reads:", sum(
    isinstance(n, ast.Attribute) and n.attr == "returncode" for n in ast.walk(fn)
))
print("caught exception types:", [
    ast.unparse(h.type) if h.type else "bare"
    for n in ast.walk(fn) if isinstance(n, ast.Try)
    for h in n.handlers
])
print("return tuple length:", next(
    len(n.value.elts) for n in ast.walk(fn)
    if isinstance(n, ast.Return) and isinstance(n.value, ast.Tuple)
))
PY

Repository: openshift/ocp-release-operator-sdk

Length of output: 31763

🌐 Web query:

govulncheck exit status non-zero vulnerabilities JSON mode official documentation

💡 Result:

When running govulncheck, the exit status behavior depends on whether you use the standard text output or a machine-readable format like JSON [1][2]. Exit Status Behavior: - Standard (Text) Mode: Govulncheck exits with code 3 if vulnerabilities are detected and code 0 if none are found [3][4]. - JSON/SARIF/OpenVEX Mode: When using -json, -format sarif, or -format openvex, govulncheck always exits with code 0, regardless of whether vulnerabilities are detected [1][2]. Official Guidance: This behavior is intentional [5]. In JSON mode, an exit status of 0 indicates that the tool executed successfully and the JSON output is trustworthy [5]. It does not indicate the absence of vulnerabilities [5]. To determine if vulnerabilities were found when using JSON mode, you must parse the resulting JSON output to check for the presence of "finding" objects [5][6]. Working with JSON Output: Govulncheck uses a streaming JSON format, which means it emits a sequence of Message objects [7][8]. Because it emits findings as it discovers them, you should process the entire stream [7][6]. If a vulnerability is found, the JSON stream will contain a Finding object [7][8]. If no Finding objects are present in the stream, no vulnerabilities were detected [6].

Citations:

Do not describe a missing govulncheck binary as a clean result.

A missing binary currently raises FileNotFoundError, so it does not produce a clean verdict. run_govulncheck also ignores non-zero exit statuses; a failed scan can pass the config and SBOM checks and return found=False. Handle command-not-found and scan failures before accepting that result.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@hack/cve-triage/README.md` around lines 174 - 175, Update the README guidance
for the callgraph step and govulncheck outcome: do not state or imply that a
missing govulncheck binary means the binary is clean. Document that
command-not-found and non-zero scan failures must be handled as failures before
accepting a run_govulncheck result, even when config and SBOM checks pass.

- If a run is interrupted, worktrees may be left behind. Check with
`git worktree list` and remove with `git worktree remove --force <path>`.
- For multi-version tickets (multiple `affectedVersion` values), the tool
analyses all versions and only closes the Jira once all pass.

---

## Known limitations

**Source tree, not shipped binary**
The analysis runs against the upstream release branch source, not the exact
container image. Downstream patches or ART modifications are not visible. For
a binary-level check use `govulncheck -mode=binary` against the extracted image.

**Reflection and unsafe**
Static call graph analysis cannot track calls made via `reflect.Value.Call`,
`unsafe.Pointer`, or dynamic function registration. In practice these patterns
are not present in the operator's critical paths.

---

## Adapting for another component

`cve_triage_core.py` contains no product-specific logic. To triage a different
binary, create a new driver by copying `triage_helm_operator_cve.py` and
updating the five constants at the top:

```python
_MODULE_PATH = "github.com/my-org/my-repo"
_CMD_NAME = "my-operator" # subdirectory under cmd/
_UPSTREAM_URL_FRAGMENT = "my-org/my-repo" # matched against git remote URLs
_KNOWN_COMPONENTS = {"openshift4/my-operator-container"}
_SANITY_DEP_PREFIXES = [ # packages always present in your binary
"sigs.k8s.io/controller-runtime",
"k8s.io/client-go",
]
```

No changes to `cve_triage_core.py` are needed.

---

## Security hardening

`cve_triage_core.py` deliberately restricts several values that would
otherwise be CLI-overridable, to keep static-analysis taint tracking (e.g.
Snyk) provably sound and to close a few real SSRF/injection/path-traversal
risks. If you're modifying this library, preserve these invariants:

| Hardening | Why |
|---|---|
| `JIRA_BASE_URL` is a hardcoded constant, not a flag/env var | Prevents pointing the tool's Basic-auth credentials at an attacker-controlled host (SSRF) |
| `_assert_allowed_url()` validates every `urlopen()` target against an allowlist (`redhat.atlassian.net`, `api.osv.dev`, `vuln.go.dev`) and rebuilds the URL from the allowlisted base | Belt-and-suspenders SSRF guard even if a caller-controlled string reaches a URL-building function |
| `JiraCreds` (email/token) is a separate dataclass from `TriageConfig` (paths/component config) | A credential value can never be conflated with the data used to build local file paths |
| `_sanitize_cve_id` / `_sanitize_vuln_id` / `_sanitize_issue_key` / `_sanitize_email` / `_sanitize_git_sha` | Reject malformed Jira/CVE/vuln-DB/git values before they're interpolated into a URL, git ref, or path |
| `_safe_output_path()` bounds-checks every report/govulncheck/callgraph output path under the intended output directory | Prevents path traversal via a crafted CVE ID or version string |
| Git refs are resolved to a 40-char commit SHA (`resolve_remote_sha` + `_sanitize_git_sha`) *before* `git worktree add`, rather than passing a branch name straight through | Breaks a taint chain from a CLI-controlled branch name through the worktree `cwd` → `go mod why` output → Jira comment → back out via `urlopen` |
| `--branch` is a no-op (branch is always derived from Jira `affectedVersion`) and `--worktree-base` is not exposed at all | Both used to be CLI-controlled values that fed directly into the worktree `cwd`, closing the same taint chain as above |
| `close_as_not_affected()` / `mark_duplicate_of_builder()` never take a `dry_run` parameter | The dry-run gate lives entirely in `run_triage` (the caller), so a CLI-tainted boolean is never in a position to guard whether the actual `jira_post()`/`urlopen()` calls execute |

---

## Agent / automation usage

When an AI agent runs this script it should follow this sequence:

1. **Always `--dry-run` first.** Confirm the branch, govulncheck output, and
analysis verdict look correct before posting to Jira.
2. **Check the exit code.** `0` = safe to proceed; `1` = investigate the error
output; `2` = the CVE is reachable and needs a code fix — do not close.
3. **Read the report.** The Markdown report in `.work/compliance/analyze-cve/`
contains the full evidence. Include its path in any follow-up communication.
4. **Run without `--dry-run` only after confirming the dry-run output.** The
tool posts a detailed Jira comment and transitions the ticket in a single
operation; there is no undo.

### Programmatic import

`cve_triage_core` can be imported directly from a Python script:

```python
import sys, os
sys.path.insert(0, "hack/cve-triage")

from cve_triage_core import (
JiraCreds, TriageConfig, resolve_credentials, detect_upstream_remote,
detect_repo_root, run_triage,
)

email, token = resolve_credentials() # reads JIRA_EMAIL / JIRA_TOKEN / files
creds = JiraCreds(jira_email=email, jira_token=token)
repo_root = detect_repo_root(os.getcwd())
remote = detect_upstream_remote(repo_root, "openshift/ocp-release-operator-sdk")

cfg = TriageConfig(
module_path="github.com/operator-framework/operator-sdk",
cmd_name="helm-operator",
known_components={"openshift4/ose-helm-operator"},
sanity_dep_prefixes=["helm.sh/helm/v3", "sigs.k8s.io/controller-runtime"],
)

exit_code = run_triage("OCPBUGS-12345", cfg, creds, repo_root, remote, dry_run=True)
sys.exit(exit_code)
```
Loading