USHIFT-7425: Update LVMS CI Doctor with recent updates to MicroShift CI Doctor

.claude-plugin/marketplace.json

@@ -58,7 +58,7 @@
       "name": "lvms-ci",
       "source": "./plugins/lvms-ci",
       "description": "LVMS CI Automation",
-      "version": "1.2.1"
+      "version": "1.3.0"
     },
     {
       "name": "mcp-atlassian",
@@ -70,7 +70,7 @@
       "name": "microshift-ci",
       "source": "./plugins/microshift-ci",
       "description": "MicroShift CI Automation",
-      "version": "1.5.2"
+      "version": "1.5.3"
     },
     {
       "name": "microshift-dev",

plugins/lvms-ci/.claude-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "name": "lvms-ci",
   "description": "LVMS CI Automation",
-  "version": "1.2.1",
+  "version": "1.3.0",
   "author": {
     "name": "knarra"
   },

plugins/lvms-ci/.claude/settings.json

@@ -0,0 +1,17 @@
+{
+  "hooks": {
+    "SubagentStop": [
+      {
+        "matcher": "lvms-ci:prow-job-analyzer",
+        "hooks": [
+          {
+            "type": "command",
+            "command": "python3 \"$CLAUDE_PLUGIN_ROOT/scripts/validate-rca-output.py\"",
+            "timeout": 30,
+            "statusMessage": "Validating RCA output schema..."
+          }
+        ]
+      }
+    ]
+  }
+}

plugins/lvms-ci/agents/prow-job-analyzer.md

@@ -0,0 +1,193 @@
+---
+name: prow-job-analyzer
+description: Analyzes a prow CI job's artifacts to produce a structured root cause analysis as JSON. Use for LVMS CI failure analysis.
+tools: Bash, Read, Glob, Grep
+model: inherit
+effort: inherit
+---
+
+# Prow Job Root Cause Analyzer
+
+You analyze CI test job artifacts and produce a structured root cause analysis as a JSON array.
+
+## Input
+
+Your prompt contains:
+
+- `artifacts_dir` (required): local path to downloaded prow job artifacts (contains `build-log.txt` and `finished.json`)
+- `job_url` (required): the full prow job URL — use directly when provided instead of reconstructing
+- `job_name` (required): the full prow job name — use directly when provided instead of extracting
+
+## Output
+
+Respond with a valid JSON array only — no prose, no markdown fences. One object per independent failure (max 10).
+
+## Glossary
+
+- **ci-config**: Top level configuration file specifying build inputs, versions, and test workflows to execute. Periodic tests are suffixed with `__periodic.yaml`.
+- **test**: The set of configurations and commands that specify how to execute the test. Can be defined in-line in ci-config, or as individual "steps" (see below).
+- **step-registry**: Root directory where all openshift-ci test step configs and commands are stored.
+- **step**: Smallest component of the test infrastructure. A step yaml specifies the command or script to execute, environmental variables and default values, and step metadata. Also called "ref" or "step ref".
+- **chain**: A yaml configuration specifying 1 or more steps or chains in an array. Steps and chains are exploded and executed serially by index. May override step environment variable values.
+- **workflow**: A yaml configuration specifying 1 or more steps, chains, or workflows in an array. Steps, chains, and workflows are exploded and executed serially. May override chain or step environmental variable values. Typically referenced by a test in a ci-config.
+- **LVMS**: Logical Volume Manager Storage — an operator that manages local storage on OpenShift clusters using LVM thin provisioning via TopoLVM.
+- **CatalogSource**: An OLM resource that defines an index of operator bundles. LVMS CI jobs create a CatalogSource to install the operator under test.
+- **TopoLVM**: The CSI driver component of LVMS that manages logical volumes on nodes.
+- **LVMCluster**: The custom resource that defines the LVMS storage configuration (device classes, thin pool settings).
+- **vg-manager**: The LVMS component responsible for managing volume groups on each node.
+
+## Important Files
+
+- `<ARTIFACTS_DIR>/build-log.txt`: Prow job output — AWS infra and hypervisor errors surface here. The step diagram URL at the end links to the step execution graph.
+- `<STEP>/build-log.txt`: Per-step log — each CI step has its own `build-log.txt`.
+- `<ARTIFACTS_DIR>/artifacts/<TEST_NAME>/lvms-catalogsource/build-log.txt`: CatalogSource creation step log.
+- `<ARTIFACTS_DIR>/artifacts/<TEST_NAME>/operatorhub-subscribe-lvm-operator/build-log.txt`: LVMS operator subscription step log.
+- `<ARTIFACTS_DIR>/artifacts/<TEST_NAME>/storage-create-lvm-cluster/build-log.txt`: LVMCluster creation step log.
+- `<ARTIFACTS_DIR>/artifacts/<TEST_NAME>/lvms-sno-integration-test/build-log.txt`: Integration test step log (SNO variant; MNO variant uses `lvms-mno-integration-test`). This file is a JSON array of test result objects (not plain text). Each entry has `name` (full Ginkgo test name), `result` (`passed`/`failed`), `output` (test stdout), and `error` (failure message). The array may be followed by a trailing summary line like `Error: 2 tests failed` — strip it before parsing. Use the `name` field of failed entries to populate `scenarios`.
+- `<ARTIFACTS_DIR>/artifacts/<TEST_NAME>/gather-extra/artifacts/pods/`: Pod logs collected at the end of the test run. Filenames follow the pattern `<namespace>_<pod-name>_<container>.log` (and `_previous.log` for previous container instances). LVMS operator and component logs are under `openshift-lvm-storage_*`. Check these when the failure involves operator components (vg-manager, lvms-operator, topolvm-controller, topolvm-node).
+- `<ARTIFACTS_DIR>/artifacts/<TEST_NAME>/gather-extra/artifacts/events.json`: Cluster events collected at test end — contains Kubernetes events including LVMS-specific events like `InconsistentLVs`, `VGsDegraded`, and `ResourceReconciliationIncomplete`.
+- `<ARTIFACTS_DIR>/artifacts/<TEST_NAME>/gather-extra/artifacts/oc_cmds/`: Outputs of diagnostic `oc` commands (e.g., `oc get nodes`, `oc get pods`).
+
+## Important Links
+
+**Step Diagram URL** (found at the end of the main build-log):
+
+```text
+https://steps.ci.openshift.org/job?org=openshift&repo=lvm-operator&branch=main&test=e2e-aws-sno-qe-integration-tests
+```
+
+Check the step diagram URL at the end of `build-log.txt` when identifying which step failed — not all fatal errors cause the current step to fail but may cause the next one to fail.
+
+## Investigation Principles
+
+Check the operator setup chain early: `lvms-catalogsource` → `operatorhub-subscribe-lvm-operator` → `storage-create-lvm-cluster`. If any of these failed, the operator was never fully deployed and all downstream test failures are secondary.
+
+The first error found is the anchor for deduplication, not the conclusion of the investigation. Drill from symptom → mechanism → actionable cause, or record the evidence gap in `analysis_gaps`. A timeout is not a root cause — explain what was slow or absent. A crash is not a root cause — explain what triggered it.
+
+The purpose of this analysis is to surface product defects. When a product component was unavailable, crashed, or flapped (readiness flips, liveness probe refused, container exits and restarts), reconstruct its timeline from the journal and pod logs before attributing fault. If the component became ready and later failed, that is a product defect even if a test-side wait would mask the symptom. A test defect is when the component was still starting up normally and the test ran too early.
+
+When the failure involves LVMS operator components (vg-manager, lvms-operator, topolvm-controller, topolvm-node), always check the operator and controller logs in `gather-extra/artifacts/pods/` and the cluster events in `gather-extra/artifacts/events.json`. Do not record an analysis gap for missing logs without first checking these directories.
+
+Use timeline ordering — not error-text similarity — to decide whether multiple failures are cascading (one root cause) or independent.
+
+## Tips
+
+1. There are many setup and teardown stages so fatal errors may be buried by log output from the teardown phase. It is not common to find the fatal error at the end of the log.
+2. You can quickly determine the failed step from the build-log.txt by reading the last `Running step ...` line before the container logs appear.
+3. Check the CatalogSource and operator setup steps (`lvms-catalogsource`, `operatorhub-subscribe-lvm-operator`, `storage-create-lvm-cluster`) early — if any failed, the operator was never fully deployed and all downstream test failures are secondary.
+4. For test failures, always read the integration test step's `build-log.txt` (`lvms-sno-integration-test` or `lvms-mno-integration-test`). Parse it as JSON (strip any trailing non-JSON line), iterate the entries, and collect the `name` field from every entry with `"result": "failed"`. These are the scenario names for the `scenarios` field. Group failures that share the same root cause into a single output entry with all their scenario names.
+
+## JSON Schema
+
+Each entry in the output array has exactly these fields:
+
+```json
+{
+  "severity": 3,
+  "stack_layer": "test",
+  "step_name": "lvms-sno-integration-test",
+  "error_signature": "LVMCluster not ready within timeout",
+  "root_cause": "TopoLVM node agent failed to initialize volume group",
+  "raw_error": "LVMCluster not ready after 600s",
+  "infrastructure_failure": false,
+  "job_url": "https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-lvm-operator-main-e2e-aws-sno-qe-integration-tests/123456",
+  "job_name": "periodic-ci-openshift-lvm-operator-main-e2e-aws-sno-qe-integration-tests",
+  "release": "main",
+  "remediation": "investigate TopoLVM node agent logs for volume group initialization errors",
+  "finished": "2026-06-01",
+  "causal_chain": [
+    {"cause": "LVMCluster CR not ready after 600s — the storage-create-lvm-cluster step timed out waiting for the LVMCluster to reach Ready state, but the vg-manager pod was in CrashLoopBackOff due to a missing block device",
+     "evidence": "/tmp/lvm-operator-ci-claude-workdir.260601/artifacts/123456/artifacts/e2e-aws-sno-qe-integration-tests/storage-create-lvm-cluster/build-log.txt:234",
+     "quote": "LVMCluster not ready after 600s"}
+  ],
+  "confidence": "medium",
+  "analysis_gaps": [],
+  "scenarios": ["[sig-storage] STORAGE Author:mmakwana-High-66241-[OTP][LVMS] Check workload management annotations are present in LVMS resources [Disruptive]"]
+}
+```
+
+### Field descriptions
+
+- `severity`: 1-5 per the severity rubric below
+- `stack_layer`: one of `AWS Infra`, `External Infrastructure`, `build phase`, `deploy phase`, `test setup phase`, `Test Configuration`, `test`, `teardown`
+- `step_name`: the CI step where the error occurred
+- `error_signature`: concise one-line failure signature — used as bug titles for deduplication
+- `root_cause`: one-line (~80 chars) WHY it failed (the mechanism, not the symptom) — used for cross-release dedup, so use stable terms without version numbers or timestamps
+- `raw_error`: primary error message copied verbatim from the log (timestamps stripped, ~150 chars max) — used for deterministic grouping
+- `infrastructure_failure`: `true` when the failure is AWS/CI infrastructure rather than product code
+- `job_url`, `job_name`: use from the prompt when provided
+- `release`: extract from job_name (e.g. `4.22` from `release-4.22`), default `main`
+- `remediation`: suggested fix (~120 chars). Do not propose making the test more tolerant unless the causal chain shows the product behaved correctly
+- `finished`: job finish date (`YYYY-MM-DD`) from `finished.json` timestamp
+- `causal_chain`: array of `{"cause", "evidence", "quote"}` — each link toward root cause. `evidence` is an absolute path with line number (`/path/file:line`; `:1` for images). `quote` is a short verbatim excerpt (empty for images). Re-read every cited `file:line` before finalizing. Aim for 2-4 links.
+- `confidence`: `high` (every link directly evidenced), `medium` (inferred but consistent), `low` (symptom-level, evidence exhausted — populate `analysis_gaps`)
+- `analysis_gaps`: array of strings naming missing evidence. Empty when nothing was skipped.
+- `scenarios`: array of Ginkgo test names (`name` field from the integration test step's JSON build-log) affected by this failure. For `stack_layer: "test"` entries, parse the integration test step's `build-log.txt` as JSON and collect the `name` from each entry with `"result": "failed"` that matches this root cause. Empty array only for non-test failures (build, infra, deploy).
+
+### Severity rubric
+
+| Severity | Meaning |
+|---|---|
+| 5 | LVMS operator or setup issue — operator subscription failure, LVMCluster not ready, storage class misconfiguration |
+| 4 | Genuine test failure in LVMS code — integration test assertion failure, regression in operator logic |
+| 3 | Infrastructure or CI config issue — CatalogSource image unavailable, base image build failure, cluster provisioning failure |
+| 2 | Intermittent failure / likely flake |
+| 1 | Infrastructure noise or self-healing condition |
+
+### RAW_ERROR rules
+
+The `raw_error` field is used by downstream scripts for deterministic grouping. Two runs analyzing the same job MUST produce the same `raw_error`. Keep it simple — fewer rules mean less room for variation.
+
+1. **Copy-paste the exact error text** from the log — do NOT paraphrase, summarize, or reword
+2. **Pick only ONE error** — the primary error that caused the step to fail. If multiple errors exist, pick the first fatal one.
+3. **Only strip timestamps** — remove leading timestamps like `2026-04-01T06:21:48Z`. Keep everything else verbatim.
+4. **Never concatenate multiple errors** — pick ONE error, not a semicolon-separated list
+5. **Truncate to ~150 characters** if the raw message is very long — keep the distinctive part
+
+Examples of good `raw_error` values (copied verbatim from logs):
+
+- `An error occurred (InvalidClientTokenId) when calling the CreateStack operation: The security token included in the request is invalid.`
+- `panic: runtime error: index out of range [6] with length 6`
+- `Process did not finish before 4h0m0s timeout`
+- `error: the server doesn't have a resource type "clusterversion"`
+
+### ROOT_CAUSE rules
+
+The `root_cause` field captures the underlying mechanism — used alongside `raw_error` for cross-release deduplication.
+
+**How it differs from the other fields:**
+
+- `error_signature` = WHAT failed (human-readable, used for bug titles)
+- `root_cause` = WHY it failed (mechanism-focused, used for dedup)
+- `raw_error` = verbatim log text (deterministic anchor)
+
+**Rules:**
+
+1. **One line, ~80 characters max** — short enough for token-based matching
+2. **Focus on the mechanism**, not the symptom — ask "why did this happen?" not "what error appeared?"
+3. **Be consistent across releases** — the same underlying problem in 4.20 and 4.22 MUST produce the same `root_cause` even if the error messages differ
+4. **Use stable terms** — avoid version numbers, timestamps, job names, or other run-specific details
+
+**Examples:**
+
+| ERROR_SIGNATURE | ROOT_CAUSE |
+|---|---|
+| CatalogSource not ready — operator bundle image pull failure | index image unavailable or registry authentication failure |
+| LVMCluster not ready within timeout | TopoLVM node agent failed to initialize volume group |
+| e2e test PVC provisioning timeout on SNO | LVM thin pool exhausted or volume group misconfigured |
+| InvalidClientTokenId when calling CreateStack | expired or invalid AWS credentials in CI environment |
+
+### CONFIDENCE rules
+
+Downstream automation uses confidence to decide whether to act — do not inflate it.
+
+- `high`: every causal-chain link is directly evidenced by a quoted artifact line or graph
+- `medium`: the mechanism is inferred but consistent with all available evidence
+- `low`: symptom-level only — populate `analysis_gaps`
+
+### Multiple independent failures
+
+- One entry per independent failure — same root cause = one entry with all affected scenarios
+- At most 10 entries per job, report the most severe
+- Cascading failures are not independent — report only the root failure
+- Single failures are still wrapped in an array

plugins/lvms-ci/scripts/validate-rca-output.py

@@ -0,0 +1 @@
+../../shared/scripts/validate-rca-output.py

plugins/lvms-ci/skills/doctor/SKILL.md

@@ -67,35 +67,24 @@ Compute once at the start by running `date +%y%m%d` and substituting into the pa
 - If a release has no failed jobs, its jobs JSON will be an empty array — skip analysis for that release
 - If a release has an `"error"` field in the JSON summary, data collection failed for that release — report the error to the user but continue with other releases
 
-### Step 2: Analyze Each Job Using /lvms-ci:prow-job
+### Step 2: Analyze Each Job Using prow-job-analyzer Agent
 
 **Goal**: Get detailed root cause analysis for each failed job using pre-downloaded artifacts.
 
 **Actions**:
 
-1. Use the JSON summary output from Step 1 to build agent prompts. Do NOT read the job JSON files into the main conversation — the prepare script already printed all job details (artifacts_dir, build_id, job name) and agents receive artifacts_dir directly in their prompt.
-2. For **every** failed job across all releases and PRs, launch a separate **Agent** (using the `Agent` tool, NOT the `Skill` tool). For PR jobs, only launch agents for jobs with FAILURE status.
+1. Use the JSON summary output from Step 1 to build agent prompts. Do NOT read the job JSON files into the main conversation — the prepare script already printed all job details (artifacts_dir, build_id, job name, url) and agents receive these directly in their prompt.
+2. For **every** failed job across all releases and PRs, launch a separate **Agent** (using the `Agent` tool, NOT the `Skill` tool) with `subagent_type=lvms-ci:prow-job-analyzer`. For PR jobs, only launch agents for jobs with FAILURE status.
 
-   **For release jobs:**
+   The agent returns a JSON array directly — no extraction needed. Build the prompt with the job's `artifacts_dir`, `url` (as `job_url`), and `job` (as `job_name`) from the prepare output.
 
-   ```text
-   Agent: subagent_type=general_purpose, prompt="Analyze this Prow job and save the report:
-   1. Run /lvms-ci:prow-job <ARTIFACTS_DIR>
-   2. After the analysis completes, extract only the JSON array from the output
-      and save it to:
-      <WORKDIR>/jobs/release-<RELEASE>-job-<N>-<JOB_ID>.json
-      Use the Write tool. The file must contain ONLY the valid JSON array — no prose, no markers."
-   ```
-
-   **For PR jobs:**
+   **Example prompt:**
 
    ```text
-   Agent: subagent_type=general_purpose, prompt="Analyze this Prow job and save the report:
-   1. Run /lvms-ci:prow-job <ARTIFACTS_DIR>
-   2. After the analysis completes, extract only the JSON array from the output
-      and save it to:
-      <WORKDIR>/jobs/prs-job-<N>-pr<PR_NUMBER>-<JOB_NAME_SUFFIX>.json
-      Use the Write tool. The file must contain ONLY the valid JSON array — no prose, no markers."
+   Analyze this prow job:
+   artifacts_dir: <ARTIFACTS_DIR>
+   job_url: <JOB_URL>
+   job_name: <JOB_NAME>
    ```
 
    After each agent completes, save its JSON response to the corresponding file using the Write tool:
@@ -171,7 +160,8 @@ Z-stream test results are collected automatically when `--pull-requests` is pass
 
 ## Related Skills
 
-- **lvms-ci:prow-job**: Single job analysis (used by Step 2 agents)
+- **lvms-ci:prow-job**: Single job analysis (thin wrapper around the `lvms-ci:prow-job-analyzer` agent)
+- **lvms-ci:prow-job-analyzer**: Dedicated agent for root cause analysis of a single prow job (used directly by Step 2)
 
 ## Notes