ci: analyze-disruption: root-cause attribution rules, onset/suspect-PR step, existing-bug discovery - #655
Open
mkowalski wants to merge 1 commit into
Open
Conversation
…R step, existing-bug discovery Fold the lessons from recent Component Readiness triage duty into the disruption analysis skill: - Step 5.6 (new): attribution rules — victim-vs-owner with upstream corroboration (dependency-error crash-loopers are usually victims, but check the provider's health before reassigning; RBAC/config faults masquerade as dependency errors), clusteroperators.json as the end-state oracle (pod existence proves nothing about etcd/apiserver serving), extract ClusterVersion/ClusterOperator condition *messages* (they name the culprit controller), read *_previous.log for any container restarted in the window, and count condition transitions (sync-loop flap vs slow rollout). - Step 7.3 (new): onset determination for new/worsening disruption — clean-vs-disrupted payload delta (incl. initial/target skew on upgrade jobs), fetch-new-prs-in-payload vetting, and the revert-PR shortcut (fresh reverts citing TRT/OCPBUGS keys yield trigger + ticket in one query); bans unanchored 'unidentified payload change' claims. - Step 7.4 (new): existing-bug discovery — fetch-related-triages with verify-before-adopt, component-scoped JIRA listing (developer vocabulary defeats keyword search), and the known recurring disruption families to rule out first. - Report templates: Root Cause Hypothesis sections now require the full causal chain, the Step 5.6 corroboration, and existing-ticket references.
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: mkowalski The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
Contributor
WalkthroughThe CI plugin version was incremented, and the disruption-analysis skill gained expanded root-cause attribution, onset analysis, known-issue checks, and updated single-run and multi-run report requirements. ChangesCI disruption guidance
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 10✅ Passed checks (10 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plugins/ci/skills/analyze-disruption/SKILL.md`:
- Around line 325-333: Revise the guidance in the provider-corroboration rule so
provider health is evidence rather than an ownership verdict. Require a
time-correlated provider failure and trace dependencies to the first independent
fault before assigning ownership; likewise, do not conclude the victim is
responsible solely because the provider appears healthy.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 0a773c56-0b45-4439-9e8d-e23376a17249
📒 Files selected for processing (2)
plugins/ci/.claude-plugin/plugin.jsonplugins/ci/skills/analyze-disruption/SKILL.md
Comment on lines
+325
to
+333
| 1. **A component whose error names a missing upstream dependency is a victim, not an owner — | ||
| but corroborate before reassigning.** If a disrupted backend's serving pods log | ||
| "cannot list/watch/reach X" (e.g., a router crash-looping on | ||
| `failed to list *v1.Route: the server could not find the requested resource`), check X's | ||
| provider directly (ClusterOperator conditions, APIService availability, endpoints) before | ||
| blaming either side: provider unhealthy → the disruption belongs upstream (in one real case, | ||
| router disruption traced through openshift-apiserver to etcd never deploying its static | ||
| pods); provider healthy → the fault is inside the "victim" after all (RBAC — a real | ||
| crash-looper failed on `configmaps is forbidden` — client config, or network reachability). |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Keep provider health as corroborating evidence, not an ownership verdict.
The provider unhealthy → disruption belongs upstream rule can misattribute shared lower-level failures or downstream symptoms. Require time-correlated provider failure and continue tracing to the first independent fault before assigning ownership; likewise, provider health alone should not prove the victim is at fault.
Proposed wording
- provider unhealthy → the disruption belongs upstream
+ provider unhealthy → supports an upstream hypothesis; continue tracing to the first independent fault📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| 1. **A component whose error names a missing upstream dependency is a victim, not an owner — | |
| but corroborate before reassigning.** If a disrupted backend's serving pods log | |
| "cannot list/watch/reach X" (e.g., a router crash-looping on | |
| `failed to list *v1.Route: the server could not find the requested resource`), check X's | |
| provider directly (ClusterOperator conditions, APIService availability, endpoints) before | |
| blaming either side: provider unhealthy → the disruption belongs upstream (in one real case, | |
| router disruption traced through openshift-apiserver to etcd never deploying its static | |
| pods); provider healthy → the fault is inside the "victim" after all (RBAC — a real | |
| crash-looper failed on `configmaps is forbidden` — client config, or network reachability). | |
| 1. **A component whose error names a missing upstream dependency is a victim, not an owner — | |
| but corroborate before reassigning.** If a disrupted backend's serving pods log | |
| "cannot list/watch/reach X" (e.g., a router crash-looping on | |
| `failed to list *v1.Route: the server could not find the requested resource`), check X's | |
| provider directly (ClusterOperator conditions, APIService availability, endpoints) before | |
| blaming either side: provider unhealthy → supports an upstream hypothesis; continue tracing to the first independent fault (in one real case, router disruption traced through openshift-apiserver to etcd never deploying its static | |
| pods); provider healthy → the fault is inside the "victim" after all (RBAC — a real | |
| crash-looper failed on `configmaps is forbidden` — client config, or network reachability). |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@plugins/ci/skills/analyze-disruption/SKILL.md` around lines 325 - 333, Revise
the guidance in the provider-corroboration rule so provider health is evidence
rather than an ownership verdict. Require a time-correlated provider failure and
trace dependencies to the first independent fault before assigning ownership;
likewise, do not conclude the victim is responsible solely because the provider
appears healthy.
openshift-ci
Bot
added
the
needs-rebase
|
PR needs rebase. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Recent AI-assisted Component Readiness triage duty for
5.0-mainsurfaced several root-cause-attribution failure modes that cost real time (regressions attached to the wrong bug, a router change blamed for an etcd defect, a duplicate bug filed for an already-fixed issue). Most of them apply directly to disruption analysis, which is symptom-chain work by nature. This PR folds those lessons intoanalyze-disruption.Related: #651 (same lessons applied to the prow-job-analysis / triage-adjacent skills); #635 (the bulk-triage skill where they were learned).
Changes
Step 5.6 (new) — Root-Cause Attribution Rules
configmaps is forbiddenRBAC bug).clusteroperators.jsonas the end-state oracle: pods Running proves nothing about whether cluster etcd/apiservers were serving.*_previous.logis mandatory reading for any container that restarted in the disruption window.Step 7.3 (new) — Onset Determination and Suspect PRs for new/worsening disruption: clean-vs-disrupted payload comparison (including initial/target skew on upgrade jobs — a real regression was pure skew between a pre-GA base and post-GA target),
fetch-new-prs-in-payloadvetting, and the revert-PR shortcut (a freshRevert "..."citing a TRT/OCPBUGS key yields trigger PR + tracking ticket + recovery ETA in one query). Unanchored "unidentified payload change" claims are banned.Step 7.4 (new) — Existing Bugs and Known Disruption Families:
fetch-related-triageswith verify-before-adopt (confidence is clustering, not root cause), component-scoped JIRA listing (owning teams write summaries in developer vocabulary that keyword search misses — the exact mechanism behind a real duplicate filing), and the recurring disruption families to rule out first (etcd/Azure disk pressure, OVN upgrade CNI gaps, single-node OVS stalls, cloud LB churn).Report templates: both Root Cause Hypothesis sections now require the full causal chain, the 5.6 corroboration evidence, and existing-ticket references.
Also bumps the
ciplugin version (0.0.74 → 0.0.75).AI-generated content: This PR was created by AI based on lessons from AI-assisted triage duty. Please verify before acting on it.
Summary by CodeRabbit
Documentation
Chores