TRT-2870: A/B/C test C (historical control): pre-#604 CI skills with current payload evals

.claude-plugin/marketplace.json

@@ -88,7 +88,7 @@
       "name": "ci",
       "source": "./plugins/ci",
       "description": "A plugin to work with OpenShift CI and analyze Prow job results",
-"version": "0.0.78",
+      "version": "0.0.81",
       "category": "ci",
       "keywords": [
         "prow",

plugins/ci/.claude-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "name": "ci",
   "description": "Tools for working with OpenShift CI and analyzing Prow job results",
-  "version": "0.0.78",
+  "version": "0.0.81",
   "author": {
     "name": "github.com/openshift-eng"
   }

plugins/ci/evals/cases/payload-analysis/README.md

@@ -0,0 +1,29 @@
+# Payload-analysis eval case index
+
+Case directory names are intentionally opaque because the eval harness exposes
+them to the model under test. Keep scenario descriptions here rather than in
+directory names.
+
+| Case | Scenario |
+| --- | --- |
+| `case-001` | OCP 4.22 nightly with both a CI-configuration regression and a product regression |
+| `case-002` | OCP 5.0 nightly whose blocking failures are all new |
+| `case-003` | OCP 5.0 CI regression requiring a cluster-network-operator NetworkPolicy revert |
+| `case-004` | OCP 4.22 payload with two independent valid revert candidates |
+| `case-005` | OCP 4.22 CI cloud-credential-operator revert |
+| `case-006` | OCP 4.22 CI HyperShift revert |
+| `case-007` | OCP 4.22 OLM revert |
+| `case-008` | OCP 5.0 nightly cluster-monitoring-operator monitoring regression |
+| `case-009` | OCP 5.0 CI HyperShift builder false-positive trap |
+| `case-010` | OCP 4.18 rejected payload with multiple independent failures |
+| `case-011` | OCP 5.0 CI infrastructure-only rejection with no valid payload candidate |
+| `case-012` | OCP 4.20 rejected payload with a consecutive failure streak |
+| `case-013` | OCP 4.20 accepted payload that still contains failures |
+| `case-014` | OCP 5.0 nightly failure caused by a step-registry infrastructure change |
+| `case-018` | Mixed true/false attribution: valid `oc#2279` recommendation alongside cross-tenant etcd evidence contamination falsely attributed to `hypershift#8871` |
+| `case-019` | Disjoint ClusterOperator intervals counted as one continuous streak, producing false `api#2920` and `api#2923` revert recommendations |
+| `case-020` | Compound GCP and Azure infrastructure causal chains falsely attributed to nearby `ovn-kubernetes#3298` and `cloud-provider-azure#164` changes |
+
+`case-015` through `case-017` are reserved by the eval additions in
+[PR #626](https://github.com/openshift-eng/ai-helpers/pull/626). That PR should
+use the same opaque directory naming before merge.

plugins/ci/evals/cases/payload-analysis/case-001-4.22-nightly-ci-config-and-product-regression/annotations.yaml → plugins/ci/evals/cases/payload-analysis/case-001/annotations.yaml

@@ -2,6 +2,7 @@ expected_phase: Rejected
 expected_failed_job_count: 3
 has_revert_candidates: true
 force_accept_expected: false
+analysis_cutoff: "2026-03-27T04:56:37Z"
 
 expected_candidates:
   - pr_url: "https://github.com/openshift/cluster-authentication-operator/pull/839"
@@ -16,6 +17,7 @@ expected_candidates:
 expected_ci_config_issues:
   - pr_url: "https://github.com/openshift/release/pull/76852"
     repo: "openshift/release"
+    min_confidence: 85
     description: "Changed vSphere VCM default lease to multi-tenant, breaking UPI workflows that require single-tenant leases"
     expected_failing_jobs:
       - "install-analysis-all"

plugins/ci/evals/cases/payload-analysis/case-001-4.22-nightly-ci-config-and-product-regression/input.yaml → plugins/ci/evals/cases/payload-analysis/case-001/input.yaml

@@ -1 +1,2 @@
 payload_tag: "4.22.0-0.nightly-2026-03-26-231124"
+payload_completed_at: "2026-03-27T04:56:37Z"

plugins/ci/evals/cases/payload-analysis/case-002-5.0-nightly-all-new-failures/annotations.yaml → plugins/ci/evals/cases/payload-analysis/case-002/annotations.yaml

@@ -2,6 +2,7 @@ expected_phase: Rejected
 expected_failed_job_count: 13
 has_revert_candidates: true
 force_accept_expected: false
+analysis_cutoff: "2026-05-30T12:55:22Z"
 
 expected_candidates:
   - pr_url: "https://github.com/openshift/origin/pull/31006"
@@ -24,9 +25,13 @@ notes: >
   same payload with 63 candidate PRs. openshift/origin#31006 ("add oc-mirror
   to extensionBinaries") caused widespread test failures across multiple job
   families — serial, techpreview-serial, metal-ipi, and potentially upgrade
-  jobs. The agent must identify this PR as the root cause despite the large
-  candidate pool (63 PRs). Key signals: cross-job correlation (same PR blamed
-  for failures in many independent jobs), error messages referencing test
-  binaries/extensions, and new failure mode not present in prior payloads.
+  jobs. The eight expected_failing_jobs above are the jobs this candidate is
+  expected to explain; the other five blocking failures have separate causes,
+  so the candidate-specific list intentionally does not equal the total
+  expected_failed_job_count. The agent must identify this PR as the root cause
+  despite the large candidate pool (63 PRs). Key signals: cross-job correlation
+  (same PR blamed for failures in many independent jobs), error messages
+  referencing test binaries/extensions, and new failure mode not present in
+  prior payloads.
   force_accept_expected is false because hours_since_baseline is 11.7h (under
   the 18h threshold) and failures are product regressions, not infrastructure.

plugins/ci/evals/cases/payload-analysis/case-002-5.0-nightly-all-new-failures/input.yaml → plugins/ci/evals/cases/payload-analysis/case-002/input.yaml

@@ -1 +1,2 @@
 payload_tag: "5.0.0-0.nightly-2026-05-30-072431"
+payload_completed_at: "2026-05-30T12:55:22Z"

plugins/ci/evals/cases/payload-analysis/case-003-5.0-ci-cno-networkpolicy-revert/annotations.yaml → plugins/ci/evals/cases/payload-analysis/case-003/annotations.yaml

@@ -1,7 +1,8 @@
 expected_phase: Rejected
-expected_failed_job_count: 2
+expected_failed_job_count: 3
 has_revert_candidates: true
 force_accept_expected: false
+analysis_cutoff: "2026-05-07T20:17:14Z"
 
 expected_candidates:
   - pr_url: "https://github.com/openshift/cluster-network-operator/pull/2959"

plugins/ci/evals/cases/payload-analysis/case-003/input.yaml

@@ -0,0 +1,2 @@
+payload_tag: "5.0.0-0.ci-2026-05-07-142711"
+payload_completed_at: "2026-05-07T20:17:14Z"

plugins/ci/evals/cases/payload-analysis/case-004-4.22-two-reverts/annotations.yaml → plugins/ci/evals/cases/payload-analysis/case-004/annotations.yaml

@@ -2,6 +2,7 @@ expected_phase: Rejected
 expected_failed_job_count: 3
 has_revert_candidates: true
 force_accept_expected: false
+analysis_cutoff: "2026-03-20T12:21:33Z"
 
 expected_candidates:
   - pr_url: "https://github.com/openshift/cluster-ingress-operator/pull/1354"

plugins/ci/evals/cases/payload-analysis/case-004-4.22-two-reverts/input.yaml → plugins/ci/evals/cases/payload-analysis/case-004/input.yaml

@@ -1 +1,2 @@
 payload_tag: "4.22.0-0.nightly-2026-03-20-053450"
+payload_completed_at: "2026-03-20T12:21:33Z"

plugins/ci/evals/cases/payload-analysis/case-005-4.22-ci-cco-revert/annotations.yaml → plugins/ci/evals/cases/payload-analysis/case-005/annotations.yaml

@@ -2,6 +2,7 @@ expected_phase: Rejected
 expected_failed_job_count: 3
 has_revert_candidates: true
 force_accept_expected: false
+analysis_cutoff: "2026-03-31T10:56:34Z"
 
 expected_candidates:
   - pr_url: "https://github.com/openshift/cloud-credential-operator/pull/978"

plugins/ci/evals/cases/payload-analysis/case-005-4.22-ci-cco-revert/input.yaml → plugins/ci/evals/cases/payload-analysis/case-005/input.yaml

@@ -1 +1,2 @@
 payload_tag: "4.22.0-0.ci-2026-03-31-050515"
+payload_completed_at: "2026-03-31T10:56:34Z"

plugins/ci/evals/cases/payload-analysis/case-006-4.22-ci-hypershift-revert/annotations.yaml → plugins/ci/evals/cases/payload-analysis/case-006/annotations.yaml

@@ -2,6 +2,7 @@ expected_phase: Rejected
 expected_failed_job_count: 2
 has_revert_candidates: true
 force_accept_expected: false
+analysis_cutoff: "2026-03-31T22:55:24Z"
 
 expected_candidates:
   - pr_url: "https://github.com/openshift/hypershift/pull/7790"

plugins/ci/evals/cases/payload-analysis/case-006-4.22-ci-hypershift-revert/input.yaml → plugins/ci/evals/cases/payload-analysis/case-006/input.yaml

@@ -1 +1,2 @@
 payload_tag: "4.22.0-0.ci-2026-03-31-170515"
+payload_completed_at: "2026-03-31T22:55:24Z"

plugins/ci/evals/cases/payload-analysis/case-007-4.22-olm-revert/annotations.yaml → plugins/ci/evals/cases/payload-analysis/case-007/annotations.yaml

@@ -2,6 +2,7 @@ expected_phase: Rejected
 expected_failed_job_count: 3
 has_revert_candidates: true
 force_accept_expected: false
+analysis_cutoff: "2026-03-19T01:17:31Z"
 
 expected_candidates:
   - pr_url: "https://github.com/openshift/operator-framework-olm/pull/1256"

plugins/ci/evals/cases/payload-analysis/case-007-4.22-olm-revert/input.yaml → plugins/ci/evals/cases/payload-analysis/case-007/input.yaml

@@ -1 +1,2 @@
 payload_tag: "4.22.0-0.nightly-2026-03-18-161724"
+payload_completed_at: "2026-03-19T01:17:31Z"

plugins/ci/evals/cases/payload-analysis/case-008-5.0-nightly-cmo-monitoring-revert/annotations.yaml → plugins/ci/evals/cases/payload-analysis/case-008/annotations.yaml

@@ -1,7 +1,8 @@
-expected_phase: ""
+expected_phase: Rejected
 expected_failed_job_count: 2
 has_revert_candidates: true
 force_accept_expected: false
+analysis_cutoff: "2026-04-28T03:07:53Z"
 
 expected_candidates:
   - pr_url: "https://github.com/openshift/cluster-monitoring-operator/pull/2814"
@@ -10,10 +11,10 @@ expected_candidates:
     expected_confidence: 95
     description: "Minimal and telemetry CP monitors change caused monitoring test regressions"
     expected_failing_jobs:
-      - "e2e-aws-ovn"
+      - "aws-ovn-techpreview-serial-1of3"
 
 notes: >
   True positive from the CMO monitoring regression series. CMO #2814
   (MON-4517: Minimal and telemetry CP monitors) introduced monitoring
-  test failures. Reverted by #2901 on 2026-04-28. Expected phase is
-  empty because the payload was not formally Rejected.
+  test failures. Reverted by #2901 on 2026-04-28. The archived snapshot records
+  the payload phase as Rejected.

plugins/ci/evals/cases/payload-analysis/case-008-5.0-nightly-cmo-monitoring-revert/input.yaml → plugins/ci/evals/cases/payload-analysis/case-008/input.yaml

@@ -1 +1,2 @@
 payload_tag: "5.0.0-0.nightly-2026-04-27-183150"
+payload_completed_at: "2026-04-28T03:07:53Z"

plugins/ci/evals/cases/payload-analysis/case-009-5.0-ci-hypershift-builder-fp/annotations.yaml → plugins/ci/evals/cases/payload-analysis/case-009/annotations.yaml

@@ -2,6 +2,7 @@ expected_phase: Rejected
 expected_failed_job_count: 1
 has_revert_candidates: false
 force_accept_expected: false
+analysis_cutoff: "2026-04-14T14:18:53Z"
 
 expected_candidates: []
 

plugins/ci/evals/cases/payload-analysis/case-009/input.yaml

@@ -0,0 +1,2 @@
+payload_tag: "5.0.0-0.ci-2026-04-14-085906"
+payload_completed_at: "2026-04-14T14:18:53Z"

plugins/ci/evals/cases/payload-analysis/case-010/annotations.yaml

@@ -0,0 +1,12 @@
+expected_phase: Rejected
+expected_failed_job_count: 4
+has_revert_candidates: false
+force_accept_expected: false
+analysis_cutoff: "2026-05-14T16:40:01Z"
+expected_candidates: []
+notes: >
+  Complex case: 4 failed blocking jobs (aws-ovn-serial, aws-ovn-single-node-upgrade,
+  aws-ovn-upgrade, and gcp-ovn-rt-upgrade-4.18-minor). Tests the skill's ability
+  to launch parallel subagents, correlate multiple failures, and identify
+  patterns across job families. Previous payload was accepted, so these may be
+  new failures.

plugins/ci/evals/cases/payload-analysis/case-010-4.18-rejected-multiple-failures/input.yaml → plugins/ci/evals/cases/payload-analysis/case-010/input.yaml

@@ -1 +1,2 @@
 payload_tag: "4.18.0-0.nightly-2026-05-14-085133"
+payload_completed_at: "2026-05-14T16:40:01Z"

plugins/ci/evals/cases/payload-analysis/case-011-5.0-ci-infra-only-no-candidates/annotations.yaml → plugins/ci/evals/cases/payload-analysis/case-011/annotations.yaml

@@ -2,6 +2,7 @@ expected_phase: Rejected
 expected_failed_job_count: 1
 has_revert_candidates: false
 force_accept_expected: false
+analysis_cutoff: "2026-05-15T00:00:06Z"
 
 expected_candidates: []
 

plugins/ci/evals/cases/payload-analysis/case-011/input.yaml

@@ -0,0 +1,2 @@
+payload_tag: "5.0.0-0.ci-2026-05-14-181709"
+payload_completed_at: "2026-05-15T00:00:06Z"

plugins/ci/evals/cases/payload-analysis/case-012-4.20-rejected-streak/annotations.yaml → plugins/ci/evals/cases/payload-analysis/case-012/annotations.yaml

@@ -2,6 +2,8 @@ expected_phase: Rejected
 expected_failed_job_count: 3
 has_revert_candidates: false
 force_accept_expected: false
+analysis_cutoff: "2026-05-13T06:47:05Z"
+expected_candidates: []
 notes: >
   Rejection streak case: this payload is part of a multi-payload rejection streak
   (4 consecutive rejections: 2026-05-11-185554, 2026-05-12-035058, 2026-05-12-130101,

plugins/ci/evals/cases/payload-analysis/case-012-4.20-rejected-streak/input.yaml → plugins/ci/evals/cases/payload-analysis/case-012/input.yaml

@@ -1 +1,2 @@
 payload_tag: "4.20.0-0.nightly-2026-05-12-225204"
+payload_completed_at: "2026-05-13T06:47:05Z"

plugins/ci/evals/cases/payload-analysis/case-013-4.20-accepted-with-failures/annotations.yaml → plugins/ci/evals/cases/payload-analysis/case-013/annotations.yaml

@@ -2,6 +2,8 @@ expected_phase: Accepted
 expected_failed_job_count: 1
 has_revert_candidates: false
 force_accept_expected: false
+analysis_cutoff: "2026-05-13T15:35:28Z"
+expected_candidates: []
 notes: >
   Edge case: payload was Accepted despite 1 failed blocking job
   (hypershift-ovn-conformance-4.20). Tests that the skill correctly handles

plugins/ci/evals/cases/payload-analysis/case-013-4.20-accepted-with-failures/input.yaml → plugins/ci/evals/cases/payload-analysis/case-013/input.yaml

@@ -1 +1,2 @@
 payload_tag: "4.20.0-0.nightly-2026-05-13-064706"
+payload_completed_at: "2026-05-13T15:35:28Z"

plugins/ci/evals/cases/payload-analysis/case-014-5.0-nightly-step-registry-infra-change/annotations.yaml → plugins/ci/evals/cases/payload-analysis/case-014/annotations.yaml

@@ -2,6 +2,7 @@ expected_phase: Rejected
 expected_failed_job_count: 4
 has_revert_candidates: true
 force_accept_expected: false
+analysis_cutoff: "2026-06-17T17:06:42Z"
 
 expected_candidates:
   - pr_url: "https://github.com/openshift/release/pull/78140"

plugins/ci/evals/cases/payload-analysis/case-014-5.0-nightly-step-registry-infra-change/input.yaml → plugins/ci/evals/cases/payload-analysis/case-014/input.yaml

@@ -1 +1,2 @@
 payload_tag: "5.0.0-0.nightly-2026-06-17-095118"
+payload_completed_at: "2026-06-17T17:06:42Z"

plugins/ci/evals/cases/payload-analysis/case-018/annotations.yaml

@@ -0,0 +1,83 @@
+expected_phase: Rejected
+expected_failed_job_count: 4
+has_revert_candidates: true
+force_accept_expected: false
+analysis_cutoff: "2026-07-23T07:44:48Z"
+
+expected_candidates:
+  - pr_url: "https://github.com/openshift/oc/pull/2279"
+    component: "cli"
+    min_confidence: 85
+    expected_confidence: 100
+    description: "oc adm upgrade recommend output changed while the TechPreview serial tests still required the removed conditions preamble"
+    expected_failing_jobs:
+      - "aws-ovn-techpreview-serial-1of3"
+      - "aws-ovn-techpreview-serial-2of3"
+
+forbidden_revert_candidates:
+  - "https://github.com/openshift/hypershift/pull/8871"
+
+required_claims:
+  - name: "cross-tenant etcd evidence contamination"
+    any_of:
+      - "cross-tenant"
+      - "aggregates away namespace"
+      - "discards namespace identity"
+  - name: "target etcd pod did not roll"
+    any_of:
+      - "restartcount=0"
+      - "restart count was 0"
+      - "target etcd-0 has no restart"
+      - "no target statefulset rollout"
+
+must_not_conclude:
+  - "HyperShift PR #8871 caused the etcd leader-change failures by rolling or restarting the target HostedCluster etcd StatefulSet"
+  - "The first payload where the aggregated HyperShift test name failed proves that PR #8871 introduced the underlying leader-election signature"
+  - "A single nearby HyperShift PR that touches etcd configuration is sufficient evidence for a revert"
+
+distractors:
+  - "PR #8871 adds TLS security-profile arguments to etcd and is temporally near a six-payload job-level failure streak"
+  - "The aggregated test reports approximately one etcd leader change in 7 of 10 runs, which superficially resembles a rollout"
+  - "The PromQL result groups by pod and job but omits namespace, making another HostedCluster's etcd-0 look like the target etcd-0"
+
+key_evidence:
+  - fact: >-
+      The target HostedCluster and its etcd-0 pod were newly created with the
+      TLS arguments present from first startup. The target pod had
+      restartCount=0, so there was no target StatefulSet rollout or restart
+      caused by PR #8871 during the test.
+    artifacts:
+      - "failed child pod artifact: periodic-ci-openshift-hypershift-release-5.0-periodics-e2e-aws-ovn-conformance/2080077700080340992/artifacts/e2e-aws-ovn-conformance/dump/artifacts/namespaces/clusters-f59953faec06f68d99b5/core/pods/etcd-0.yaml"
+      - "failed child etcd log: periodic-ci-openshift-hypershift-release-5.0-periodics-e2e-aws-ovn-conformance/2080077700080340992/artifacts/e2e-aws-ovn-conformance/dump/artifacts/namespaces/clusters-f59953faec06f68d99b5/core/pods/logs/etcd-0-etcd.log"
+  - fact: >-
+      The external-topology etcd monitor query selects every management-cluster
+      namespace matching clusters-.* and then aggregates away namespace. A
+      leader election from another concurrent HostedCluster can therefore be
+      attributed to the target pod name etcd-0.
+    artifacts:
+      - "openshift/origin test/extended/etcd/leader_changes.go"
+      - "5.0/nightly/5.0.0-0.nightly-2026-07-22-233611/jobs/blocking/aggregated-hypershift-ovn-conformance-5.0/junit/results.json"
+  - fact: >-
+      PR #2279 has direct code-to-assertion evidence in two independent
+      TechPreview serial shards, and its revert PR #2320 merged before this
+      payload completed.
+    artifacts:
+      - "5.0/nightly/5.0.0-0.nightly-2026-07-22-072716/cli, cli-artifacts, deployer, tools/prs/2279/code.diff"
+      - "5.0/nightly/5.0.0-0.nightly-2026-07-22-233611/jobs/blocking/aws-ovn-techpreview-serial-1of3/job.json"
+      - "5.0/nightly/5.0.0-0.nightly-2026-07-22-233611/jobs/blocking/aws-ovn-techpreview-serial-2of3/job.json"
+
+discriminating_signal: >-
+  A causal rollout requires a target-pod lifecycle change. The target etcd-0
+  has no restart and was born with the new arguments, while the metric query
+  crosses HostedCluster namespaces and discards namespace identity. Keep the
+  independently proven oc #2279 recommendation, but reject HyperShift #8871.
+
+notes: >
+  Mixed true-positive and false-positive case. The original Payload Agent
+  correctly recommended openshift/oc#2279 at 100 for two TechPreview serial
+  failures, but also recommended openshift/hypershift#8871 at 95 for the
+  aggregated HyperShift failure. The latter attribution is false: it substitutes
+  job-level temporal onset for the underlying signature onset and never proves
+  the claimed target etcd rollout. The true defect is cross-tenant evidence
+  contamination in the Origin etcd monitor query. Source Payload Agent analysis:
+  https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/logs/periodic-ci-openshift-release-main-claude-payload-agent/2080077739515187200/artifacts/claude-payload-agent/openshift-claude-payload-agent/artifacts/payload-analysis-5.0.0-0.nightly-2026-07-22-233611-summary.html

plugins/ci/evals/cases/payload-analysis/case-018/input.yaml

@@ -0,0 +1,2 @@
+payload_tag: "5.0.0-0.nightly-2026-07-22-233611"
+payload_completed_at: "2026-07-23T07:44:48Z"