OCPQUAL-30: Pull Request Risk Analysis Agent

.gitignore

@@ -43,6 +43,9 @@ venv/
 # Working directories for skills
 .work/
 
+# Eval run outputs
+eval/runs/
+
 # Various logs and metrics tracking info
 *.log
 .anonymous_id

plugins/ci/.claude-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "name": "ci",
   "description": "Tools for working with OpenShift CI and analyzing Prow job results",
-  "version": "0.0.71",
+  "version": "0.0.72",
   "author": {
     "name": "github.com/openshift-eng"
   }

plugins/ci/agents/pr-risk-analyst.md

@@ -0,0 +1,23 @@
+---
+name: pr-risk-analyst
+description: Assess PR risk level and recommend testing strategy for OpenShift CI pull requests.
+model: sonnet
+color: warning
+---
+
+You are a PR risk analyst for OpenShift CI. Your job is to evaluate pull requests, score their risk, and recommend appropriate CI jobs to test.
+
+When the user provides a PR URL, use the `ci:assess-pr-risk` skill to perform the analysis.
+
+## Skills
+
+| Skill | Purpose |
+|-------|---------|
+| `ci:assess-pr-risk` | Score a PR's risk and recommend which e2e and payload jobs to run |
+
+A test result review skill is planned for a future iteration.
+
+## General Constraints
+
+- **Never approve or merge a PR.** You only assess risk and recommend. The human decides.
+- **You do not know what happened after a PR was merged.** Analyze presubmit results, payload job results, reviewer comments, and everything that occurred while the PR was open. But you have no knowledge of whether the PR was later reverted or caused any post-merge issues. Do not search for reverts of the specific PR under analysis, do not mention revert PRs or Jira tickets related to its post-merge outcome, and do not frame your analysis as a "retrospective" or "case study." Your report must read as a forward-looking risk assessment written at merge time. If you discover post-merge revert information incidentally, ignore it completely.

plugins/ci/evals/cases/assess-pr-risk/case-001-olm-hashstructure-upgrade/annotations.yaml

@@ -0,0 +1,28 @@
+description: |
+  OLM hashstructure v1→v2 library upgrade plus Kubernetes vendor bump and TLS changes.
+  The hashstructure upgrade fundamentally changed hashing behavior in operator reconciliation.
+  This is a known-bad PR that was later reverted — the skill must NOT know that.
+
+expected_tier: critical
+expected_score_range:
+  min: 71
+  max: 100
+
+must_recommend_jobs:
+  - e2e-gcp-ovn
+  - e2e-gcp-olm
+  - e2e-gcp-console-olm
+  - e2e-upgrade
+  - e2e-aws-upgrade-ovn-single-node
+  - e2e-aws-olmv0-ext
+  - e2e-aws-ovn-hypershift
+
+must_not_recommend_jobs:
+  - microshift
+  - vsphere
+
+must_skip_jobs: []
+
+expected_hotspots:
+  - hypershift
+  - upgrade

plugins/ci/evals/cases/assess-pr-risk/case-001-olm-hashstructure-upgrade/input.yaml

@@ -0,0 +1 @@
+pr_url: https://github.com/openshift/operator-framework-olm/pull/1330

plugins/ci/evals/cases/assess-pr-risk/case-002-origin-bug-id-swap/annotations.yaml

@@ -0,0 +1,18 @@
+description: |
+  Trivial 2-line change replacing one bug ID with another in a known-flake exception
+  and a test comment. Zero functional impact. No e2e jobs should be recommended —
+  this is the baseline case for cost savings on safe PRs.
+
+expected_tier: low
+expected_score_range:
+  min: 0
+  max: 20
+
+must_recommend_jobs: []
+
+must_not_recommend_jobs:
+  - e2e
+
+must_skip_jobs: []
+
+expected_hotspots: []

plugins/ci/evals/cases/assess-pr-risk/case-002-origin-bug-id-swap/input.yaml

@@ -0,0 +1 @@
+pr_url: https://github.com/openshift/origin/pull/31350

plugins/ci/evals/cases/assess-pr-risk/case-003-origin-ote-bump/annotations.yaml

@@ -0,0 +1,26 @@
+description: |
+  Vendor bump of openshift-tests-extension to fix klog stdout contamination
+  breaking OTE binary JSON parsing. Affects metal nightly jobs specifically.
+  475 additions but mostly vendored code. Core change is in
+  pkg/test/extensions/binary.go (JSON extraction logic) with a new test file.
+  Medium risk — touches test infrastructure plumbing, not product code.
+
+expected_tier: medium
+expected_score_range:
+  min: 21
+  max: 45
+
+must_recommend_jobs:
+  - e2e
+
+must_not_recommend_jobs:
+  - upgrade
+  - vsphere
+  - hypershift
+  - csi
+  - kubevirt
+  - single-node
+
+must_skip_jobs: []
+
+expected_hotspots: []

plugins/ci/evals/cases/assess-pr-risk/case-003-origin-ote-bump/input.yaml

@@ -0,0 +1 @@
+pr_url: https://github.com/openshift/origin/pull/31349

plugins/ci/evals/cases/assess-pr-risk/eval.md

@@ -0,0 +1,65 @@
+---
+# Auto-generated by /eval-analyze — edit to override
+skill: ci:assess-pr-risk
+analyzed_at: 2026-07-20T12:00:00Z
+skill_hash: 172a0fdc73fb
+
+execution_mode: case
+headless: true
+dry_run: false
+
+suggested_judges:
+  - name: state_file_valid
+    type: check
+    description: "Validates JSON state file exists with required schema"
+  - name: risk_tier_correct
+    type: check
+    description: "Checks assessed tier matches expected tier from annotations"
+  - name: risk_score_in_range
+    type: check
+    description: "Checks risk score falls within expected annotation range"
+  - name: job_recommendations_correct
+    type: check
+    description: "Validates must_recommend and must_not_recommend job constraints"
+  - name: report_quality
+    type: llm
+    description: "LLM assessment of report completeness, justification quality, and calibration"
+  - name: budget_check
+    type: builtin
+    description: "Ensures execution cost stays under budget"
+---
+
+## Skill Analysis
+
+`ci:assess-pr-risk` takes a GitHub PR URL and produces a structured risk assessment. It gathers PR metadata, diff, comments, and repo history via `gh` CLI, then scores risk across four categories (repo profile 0-25, PR complexity 0-25, code risk factors 0-30, historical risk 0-20) for a total 0-100 score. Based on the score, it classifies risk tier (low/medium/high/critical), recommends which e2e CI jobs to run or skip, estimates testing costs, and checks for hotspot patterns (HyperShift, SNO, upgrades, IPv6).
+
+## Inputs
+
+Each test case has:
+- `input.yaml`: Single field `pr_url` — a real GitHub PR URL (e.g., `https://github.com/openshift/origin/pull/31350`). The PR must exist and be accessible via `gh`.
+- `annotations.yaml`: Expected outcomes for scoring — `expected_tier`, `expected_score_range` (min/max), `must_recommend_jobs`, `must_not_recommend_jobs`, `must_skip_jobs`, `expected_hotspots`.
+
+## Outputs
+
+The skill writes a single JSON state file to `.work/pr-risk/<org>-<repo>-<pr_number>.json` containing the full assessment: PR metadata, risk score and breakdown, jobs to run/skip, coverage gaps, key risks, and cost estimates. It also produces a markdown report in conversation output.
+
+## Pipeline Flow
+
+1. **Gather PR Data** — `gh pr view`, `gh pr diff`, PR comments (including CodeRabbit), recent reverts
+2. **Calculate Risk Score** — four categories scored and summed
+3. **Determine Risk Tier** — score mapped to low/medium/high/critical
+4. **Recommend Testing** — fetch repo CI config from openshift/release, query Sippy for job durations, classify e2e jobs, check hotspots, format recommendations
+5. **Estimate Costs** — write job decisions to JSON, run `estimate_cost.py`
+6. **Write State File** — JSON to `.work/pr-risk/`
+7. **Produce Report** — structured markdown output
+
+Sub-skills: `ci:prow-job-cost-estimator` for cost estimation. Helper script: `fetch_jobs.py` for Sippy queries.
+
+## Quality Criteria
+
+- Risk tier and score align with PR characteristics (trivial changes score low, API/operator changes score high)
+- Job recommendations are calibrated: low-risk PRs skip e2e, high-risk PRs include upgrade/hypershift/SNO jobs
+- Forbidden job types (e.g., vSphere for non-vsphere PRs) are not recommended
+- Report includes specific code references justifying scores, not generic statements
+- Cost estimate is present and uses the cost estimator script output
+- State JSON file is well-formed with all required fields