USHIFT-7420: Enable CI Doctor to use JSON based PCP graphs

.claude-plugin/marketplace.json

@@ -70,7 +70,7 @@
       "name": "microshift-ci",
       "source": "./plugins/microshift-ci",
       "description": "MicroShift CI Automation",
-      "version": "1.5.3"
+      "version": "1.5.4"
     },
     {
       "name": "microshift-dev",

plugins/microshift-ci/.claude-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "name": "microshift-ci",
   "description": "MicroShift CI Automation",
-  "version": "1.5.3",
+  "version": "1.5.4",
   "author": {
     "name": "ggiguash"
   },

plugins/microshift-ci/agents/prow-job-analyzer.md

@@ -17,7 +17,7 @@ Your prompt contains:
 - `artifacts_dir` (required): local path to downloaded prow job artifacts (contains `build-log.txt` and `finished.json`)
 - `job_url` (required): the full prow job URL — use directly when provided instead of reconstructing
 - `job_name` (required): the full prow job name — use directly when provided instead of extracting
-- `graphs_dir` (optional): path to pre-generated PCP performance graph PNGs
+- `graphs_dir` (optional): path to pre-generated PCP performance metric JSON files
 - `source_dir` (optional): path to MicroShift source checkout
 
 ## Output
@@ -36,7 +36,7 @@ Two `Created container` events for the same pod means the first instance died. R
 
 Journal files (`journal_*.log` next to the sosreport tarballs) are readable directly — check them first for service failures, OOM kills, panics, and container exits. Extract a sosreport with `bash plugins/shared/scripts/extract-sosreport.sh <tarball>` when the investigation requires pod/container logs, including crashes, restarts, readiness flaps, or repeated container creation — pod and container logs (especially `previous.log`) exist exclusively inside the tarball. Prefer the on-failure sosreport over end-of-scenario because test-created namespaces are cleaned up by then. Match sosreport to failure by timestamp.
 
-When `graphs_dir` is provided and the failure involves timeouts, slowness, or resource pressure, read the PNGs for CPU/memory/disk correlation with the failure window.
+When `graphs_dir` is provided and the failure involves timeouts, slowness, or resource pressure, read the JSON metric files (`cpu.json`, `mem.json`, `io.json`, `disk.json`) for CPU/memory/disk/IO correlation with the failure window. Look for sustained patterns (4+ consecutive samples), not isolated spikes.
 
 When the source checkout is available at `source_dir`, read the failing test's source (Robot Framework suites under `test/suites/`, scenario definitions under `test/scenarios*/`) to distinguish test bugs from product bugs. If absent, note it in `analysis_gaps`.
 
@@ -75,8 +75,8 @@ Each entry in the output array has exactly these fields:
      "evidence": "/tmp/microshift-ci-claude-workdir.260601/artifacts/123456/artifacts/e2e-aws-tests-arm-nightly/openshift-microshift-e2e-metal-tests/artifacts/scenario-info/el96-lrel@standard1/rf-debug.log:2241",
      "quote": "cert-manager webhook not ready after 600s"},
     {"cause": "image pulls saturated disk I/O during the startup window, delaying all service startups including cert-manager — write await exceeded 800ms for 6 consecutive minutes",
-     "evidence": "/tmp/microshift-ci-claude-workdir.260601/graphs/123456/3_disk_io.png:1",
-     "quote": ""}
+     "evidence": "/tmp/microshift-ci-claude-workdir.260601/graphs/123456/io.json:42",
+     "quote": "\"await\": [823.5,"}
   ],
   "confidence": "medium",
   "analysis_gaps": [],
@@ -97,7 +97,7 @@ Each entry in the output array has exactly these fields:
 - `release`: extract from job_name (e.g. `4.22` from `release-4.22`), default `main`
 - `remediation`: suggested fix (~120 chars). Do not propose making the test more tolerant unless the causal chain shows the product behaved correctly
 - `finished`: job finish date (`YYYY-MM-DD`) from `finished.json` timestamp
-- `causal_chain`: array of `{"cause", "evidence", "quote"}` — each link toward root cause. `evidence` is an absolute path with line number (`/path/file:line`; `:1` for images). `quote` is a short verbatim excerpt (empty for images). Re-read every cited `file:line` before finalizing. Aim for 2-4 links.
+- `causal_chain`: array of `{"cause", "evidence", "quote"}` — each link toward root cause. `evidence` is an absolute path with line number (`/path/file:line`; `:1` for binary files). `quote` is a short verbatim excerpt (empty for binary files). Re-read every cited `file:line` before finalizing. Aim for 2-4 links.
 - `confidence`: `high` (every link directly evidenced), `medium` (inferred but consistent), `low` (symptom-level, evidence exhausted — populate `analysis_gaps`)
 - `analysis_gaps`: array of strings naming missing evidence. Empty when nothing was skipped.
 - `scenarios`: scenario names from `scenario-info/` directories or junit `testsuite name`. Empty array for non-scenario failures.
@@ -136,7 +136,7 @@ One line, ~80 chars. Focus on the mechanism. Use stable terms — the same under
 
 Downstream automation uses confidence to decide whether to act — do not inflate it.
 
-- `high`: every causal-chain link is directly evidenced by a quoted artifact line or graph
+- `high`: every causal-chain link is directly evidenced by a quoted artifact line or metric data point
 - `medium`: the mechanism is inferred but consistent with all available evidence
 - `low`: symptom-level only — populate `analysis_gaps`
 

plugins/microshift-ci/scripts/pcp-graphs/README.md

@@ -1,8 +1,9 @@
-# PCP Performance Graphs for MicroShift CI
+# PCP Performance Metrics for MicroShift CI
 
-Generate performance graphs from PCP (Performance Co-Pilot) archives
+Extract performance metrics from PCP (Performance Co-Pilot) archives
 collected during MicroShift CI job runs. Produces CPU, memory, disk I/O,
-and disk usage charts that are embedded in the ci-doctor HTML report.
+and disk usage JSON files that are embedded as interactive Chart.js charts
+in the ci-doctor HTML report.
 
 ## Background
 
@@ -47,13 +48,12 @@ The directory contains files like `yyyymmdd.hh.mm.{0,index,meta}` and a
 ## Prerequisites
 
 - `pcp-export-pcp2json` package (provides the `pcp2json` command)
-- Python 3 with `matplotlib`
+- Python 3
 
 Install:
 
 ```bash
 sudo dnf install -y pcp-export-pcp2json
-pip install matplotlib
 ```
 
 ## Usage
@@ -64,42 +64,40 @@ Graphs are generated automatically by `doctor.sh graphs`:
 bash doctor.sh graphs --workdir /tmp/microshift-ci-claude-workdir.YYMMDD
 ```
 
-This finds all PCP archives in downloaded artifacts and produces PNG
-graphs at `${WORKDIR}/graphs/<build_id>/`. The `finalize` step then
-embeds them as base64 in the HTML report with a tabbed UI per job.
+This finds all PCP archives in downloaded artifacts and produces JSON
+metric files at `${WORKDIR}/graphs/<build_id>/`. The `finalize` step
+embeds the JSON data in the HTML report as interactive Chart.js charts.
 
 ## Output
 
 | File | Description |
 |---|---|
-| `1_cpu_usage.png` | CPU usage stacked area: User (blue), I/O Wait (orange), System (red) |
-| `2_mem_usage.png` | Memory usage stacked area: Used (red), Cached (orange), Total (dashed) |
-| `3_disk_io.png` | Disk I/O chart: Read OPS (blue), Write OPS (red), Await (green dashed) |
-| `4_disk_usage.png` | Disk usage by partition: fill % per mount point |
-
-Numeric prefixes control tab display order in the HTML report.
+| `cpu.json` | CPU usage: user%, sys%, iowait%, idle% at 15s intervals |
+| `mem.json` | Memory usage: used_gb, cached_gb, free_gb, total_gb at 15s intervals |
+| `io.json` | Disk I/O: read/write ops/s, iops, await (ms), queue depth at 15s intervals |
+| `disk.json` | Disk usage per partition: used_pct%, used_gb at 15s intervals |
 
 ## Files
 
 | File | Purpose |
 |---|---|
-| `generate-graphs.sh` | Orchestrator: finds PCP archives, runs extraction and plotting in parallel |
+| `generate-graphs.sh` | Orchestrator: finds PCP archives, runs extraction in parallel |
 | `extract_cpu.sh` | Runs `pcp2json` for CPU metrics, pipes through `parse_cpu.py` |
 | `parse_cpu.py` | Parses pcp2json CPU output (user, sys, iowait, idle), normalizes to percentages |
-| `plot_cpu.py` | Generates CPU usage PNG with stacked area chart and peak table |
 | `extract_mem.sh` | Runs `pcp2json` for memory metrics, pipes through `parse_mem.py` |
 | `parse_mem.py` | Parses pcp2json memory output (used, free, cached, physmem), converts to GB |
-| `plot_mem.py` | Generates memory usage PNG with stacked area chart |
 | `extract_io.sh` | Runs `pcp2json` for disk metrics, pipes through `parse_pcp.py` |
 | `parse_pcp.py` | Parses pcp2json disk output, aggregates per-device (sum read/write, max await) |
-| `plot_io.py` | Generates disk I/O PNG with dual Y-axes (OPS + await) |
 | `extract_disk_usage.sh` | Runs `pcp2json` for filesystem metrics, pipes through `parse_disk_usage.py` |
 | `parse_disk_usage.py` | Parses pcp2json filesys output, tracks all partitions as usage percentages |
-| `plot_disk_usage.py` | Generates per-partition disk usage PNG with mount points in legend |
+| `pcp-charts.js` | Shared Chart.js rendering functions for CPU/mem/io/disk charts |
+| `create-pcp-dashboard.py` | Standalone interactive HTML dashboard from PCP metrics |
+| `generate-dashboard.sh` | Per-scenario dashboard generator (handles per-VM PCP tarballs) |
+| `plot_*.py` | Legacy matplotlib plotters (no longer called by the pipeline) |
 
-## Adding a New Graph Type
+## Adding a New Metric Type
 
 1. Create `extract_<type>.sh` and `parse_<type>.py` (follow existing patterns)
-2. Create `plot_<type>.py`
-3. Add a block to `generate-graphs.sh` with the next numeric prefix (e.g. `5_`)
-4. No changes needed in `create-report.py` — it auto-discovers all `*.png` files
+2. Add a block to `generate-graphs.sh` to extract the new metric
+3. Add a rendering function to `pcp-charts.js`
+4. Update `create-report.py` to load the new JSON file in `_METRIC_FILES`

plugins/microshift-ci/skills/doctor/SKILL.md

@@ -78,12 +78,12 @@ Compute once at the start by running `date +%y%m%d` and substituting into the pa
    bash plugins/microshift-ci/scripts/doctor.sh graphs --component microshift --workdir <WORKDIR>
    ```
 
-2. The script finds PCP archives in downloaded artifacts and generates PNG graphs at `<WORKDIR>/graphs/<build_id>/`:
-   - `1_cpu_usage.png` — CPU usage (user, system, I/O wait)
-   - `2_mem_usage.png` — Memory usage (used, cached)
-   - `3_disk_io.png` — Disk I/O (read/write OPS, await)
-   - `4_disk_usage.png` — Disk usage by partition (% fill)
-3. If prerequisites are missing (`pcp2json`, `matplotlib`), the script errors and stops.
+2. The script finds PCP archives in downloaded artifacts and generates JSON metric files at `<WORKDIR>/graphs/<build_id>/`:
+   - `cpu.json` — CPU usage (user, system, I/O wait, idle)
+   - `mem.json` — Memory usage (used, cached, free, total)
+   - `io.json` — Disk I/O (read/write ops, await, queue depth)
+   - `disk.json` — Disk usage by partition (% fill, GB used)
+3. If prerequisites are missing (`pcp2json`), the script errors and stops.
 
 ### Step 2: Analyze Each Job Using microshift-ci:prow-job-analyzer Agent
 
@@ -217,8 +217,7 @@ HTML report generated: <WORKDIR>/report-microshift-ci-doctor.html
 - MCP Jira server must be configured (for bug correlation)
 - Internet access to fetch job data from Prow/GCS
 - Bash shell, Python 3
-- `pcp-export-pcp2json` — for PCP graph generation
-- `matplotlib` Python package — for PCP graph plotting
+- `pcp-export-pcp2json` — for PCP metric extraction
 
 ## Related Skills