USHIFT-7335: Trim prow-job-analyzer prose and make it less prescriptive - #239
Conversation
openshift-ci
Bot
added
the
do-not-merge/work-in-progress
|
Skipping CI for Draft Pull Request. |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: pmtk The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
Bot
added
the
approved
|
Warning Review limit reached
Next review available in: 31 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Repository YAML (base), Central YAML (inherited) Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (4)
WalkthroughThe Prow job analyzer prompt now enforces JSON-only output and tighter investigation, evidence, schema, and failure-classification rules. The MicroShift CI primer documents additional Prow evidence sources and revises related guidance. Plugin metadata versions are updated to 1.5.2. ChangesMicroShift CI guidance
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 10 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (10 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
plugins/microshift-ci/agents/prow-job-analyzer.md (1)
25-147: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winRun the repository markdownlint flow before committing.
Per CONTRIBUTING.md and the Markdown coding guidelines, lint the Markdown files with the repo’s configured
markdownlint-cli2invocation (scripts/lint-markdown.shor the configured.markdownlint-cli2.jsonccommand), not onlynpx markdownlint-cli2 '**/*.md'in this environment.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/microshift-ci/agents/prow-job-analyzer.md` around lines 25 - 147, Run the repository-configured Markdown lint flow for the updated agents documentation before committing, using scripts/lint-markdown.sh or the command specified by .markdownlint-cli2.jsonc rather than a standalone npx markdownlint-cli2 invocation.Sources: Coding guidelines, Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plugins/microshift-ci/agents/prow-job-analyzer.md`:
- Line 57: Update the validator in validate-rca-output.py to enforce the
documented output contract: reject entries containing keys outside the specified
fields, and require both analysis_gaps and scenarios to be arrays rather than
null. Keep validation of all existing required fields unchanged.
- Line 37: Update the sosreport workflow guidance near the journal-first
instructions so extraction is not limited to journal-detected crashes or
restarts. Require extracting the relevant sosreport whenever pod/container
evidence is needed, including product-versus-test diagnosis or inspecting
previous.log after multiple container creations, while retaining journal files
as the initial source and preferring the on-failure report matched by timestamp.
In `@plugins/microshift-ci/agents/references/microshift-ci-primer.md`:
- Around line 148-149: Correct the artifact-path documentation around the
artifact table in the microshift-ci primer: clarify that
<ARTIFACTS_DIR>/build-log.txt is located at the artifact root, while
<STEP>/build-log.txt remains under the scenario-info/<scenario>/ step paths.
Update the surrounding scope statement so it does not incorrectly apply the
scenario directory prefix to the root-level entry.
---
Outside diff comments:
In `@plugins/microshift-ci/agents/prow-job-analyzer.md`:
- Around line 25-147: Run the repository-configured Markdown lint flow for the
updated agents documentation before committing, using scripts/lint-markdown.sh
or the command specified by .markdownlint-cli2.jsonc rather than a standalone
npx markdownlint-cli2 invocation.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Pro Plus
Run ID: 63f13b8f-5518-4a27-b927-3ba7661e4d24
📒 Files selected for processing (2)
plugins/microshift-ci/agents/prow-job-analyzer.mdplugins/microshift-ci/agents/references/microshift-ci-primer.md
e719d23 to
b5cd2a2
Compare
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plugins/microshift-ci/agents/prow-job-analyzer.md`:
- Line 25: Clarify the prompt’s no-failure behavior as a valid empty JSON array,
then update validate_json_text in validate-rca-output.py to accept [] while
rejecting arrays with more than 10 entries. Add positive tests for empty and
within-cap arrays and a negative test for arrays exceeding the cap, covering the
changed validation rules.
- Around line 92-94: Update the `error_signature` field description in the
schema documentation to call it a concise “failure signature” rather than a
root-cause description, while preserving its role as the observed failure used
for bug titles and deduplication. Keep `root_cause` explicitly focused on the
underlying failure mechanism.
- Line 33: Update the product-defect classification guidance in the analysis
purpose section to avoid treating every post-readiness failure as a product
defect. Require corroborating journal and pod-log evidence that the component
itself failed, while excluding failures caused by external infrastructure,
dependencies, or expected teardown before setting infrastructure_failure and
stack_layer.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Pro Plus
Run ID: 1b440f26-5168-4b9d-95a8-ecb6efcdcf34
📒 Files selected for processing (4)
.claude-plugin/marketplace.jsonplugins/microshift-ci/.claude-plugin/plugin.jsonplugins/microshift-ci/agents/prow-job-analyzer.mdplugins/microshift-ci/agents/references/microshift-ci-primer.md
🚧 Files skipped from review as they are similar to previous changes (1)
- plugins/microshift-ci/agents/references/microshift-ci-primer.md
b5cd2a2 to
db2141f
Compare
openshift-ci
Bot
added
the
ready-for-human-review
openshift-ci
Bot
removed
the
do-not-merge/work-in-progress
Restructure the agent definition from ~275 to ~148 lines (46% reduction). Research shows instruction compliance decays exponentially with count and ~65% of typical prompt instructions are redundant — specifying them consumes instruction-following capacity and reduces output predictability. Key changes: - Replace 4-step prescriptive workflow with 7 investigation principles - Remove Glossary, Important Files, Important Links, Tips sections - Eliminate content duplicated with the primer reference file (SOS Report section, journal patterns, container restart detection) - Consolidate sosreport extraction from 3 mentions to 1 - Reduce RAW_ERROR examples from 5 to 3 - Cut ROOT_CAUSE "How it differs" prose (contrastive table teaches this) - Replace aggressive emphasis (MUST/NEVER/CRITICAL) with calm directives All 16 JSON output fields retained — schema, example, severity rubric, and downstream-critical rules (RAW_ERROR determinism, ROOT_CAUSE consistency, CONFIDENCE calibration) are preserved. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Absorb Important Files entries (build-log.txt, step diagram URL)
from the analyzer into the primer's evidence table — this is the
reference file, so file-location info belongs here
- Remove SKIP_SOS paragraph (development-only detail, irrelevant to
CI artifact analysis)
- Rephrase behavioral instruction in journal section to factual
reference ("supports fault attribution" vs "reconstruct before
attributing") — behavioral guidance belongs in the agent definition,
not the reference file
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
db2141f to
f175a53
Compare
|
/lgtm |
2 participants
Summary by CodeRabbit