diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index 0d27886d..f5340093 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -70,7 +70,7 @@ "name": "microshift-ci", "source": "./plugins/microshift-ci", "description": "MicroShift CI Automation", - "version": "1.5.3" + "version": "1.5.4" }, { "name": "microshift-dev", diff --git a/plugins/microshift-ci/.claude-plugin/plugin.json b/plugins/microshift-ci/.claude-plugin/plugin.json index f4b4acda..065ede04 100644 --- a/plugins/microshift-ci/.claude-plugin/plugin.json +++ b/plugins/microshift-ci/.claude-plugin/plugin.json @@ -1,7 +1,7 @@ { "name": "microshift-ci", "description": "MicroShift CI Automation", - "version": "1.5.3", + "version": "1.5.4", "author": { "name": "ggiguash" }, diff --git a/plugins/microshift-ci/agents/prow-job-analyzer.md b/plugins/microshift-ci/agents/prow-job-analyzer.md index 9c00d92e..5a68ab2c 100644 --- a/plugins/microshift-ci/agents/prow-job-analyzer.md +++ b/plugins/microshift-ci/agents/prow-job-analyzer.md @@ -17,7 +17,7 @@ Your prompt contains: - `artifacts_dir` (required): local path to downloaded prow job artifacts (contains `build-log.txt` and `finished.json`) - `job_url` (required): the full prow job URL — use directly when provided instead of reconstructing - `job_name` (required): the full prow job name — use directly when provided instead of extracting -- `graphs_dir` (optional): path to pre-generated PCP performance graph PNGs +- `graphs_dir` (optional): path to pre-generated PCP performance metric JSON files - `source_dir` (optional): path to MicroShift source checkout ## Output @@ -36,7 +36,7 @@ Two `Created container` events for the same pod means the first instance died. R Journal files (`journal_*.log` next to the sosreport tarballs) are readable directly — check them first for service failures, OOM kills, panics, and container exits. Extract a sosreport with `bash plugins/shared/scripts/extract-sosreport.sh ` when the investigation requires pod/container logs, including crashes, restarts, readiness flaps, or repeated container creation — pod and container logs (especially `previous.log`) exist exclusively inside the tarball. Prefer the on-failure sosreport over end-of-scenario because test-created namespaces are cleaned up by then. Match sosreport to failure by timestamp. -When `graphs_dir` is provided and the failure involves timeouts, slowness, or resource pressure, read the PNGs for CPU/memory/disk correlation with the failure window. +When `graphs_dir` is provided and the failure involves timeouts, slowness, or resource pressure, read the JSON metric files (`cpu.json`, `mem.json`, `io.json`, `disk.json`) for CPU/memory/disk/IO correlation with the failure window. Look for sustained patterns (4+ consecutive samples), not isolated spikes. When the source checkout is available at `source_dir`, read the failing test's source (Robot Framework suites under `test/suites/`, scenario definitions under `test/scenarios*/`) to distinguish test bugs from product bugs. If absent, note it in `analysis_gaps`. @@ -75,8 +75,8 @@ Each entry in the output array has exactly these fields: "evidence": "/tmp/microshift-ci-claude-workdir.260601/artifacts/123456/artifacts/e2e-aws-tests-arm-nightly/openshift-microshift-e2e-metal-tests/artifacts/scenario-info/el96-lrel@standard1/rf-debug.log:2241", "quote": "cert-manager webhook not ready after 600s"}, {"cause": "image pulls saturated disk I/O during the startup window, delaying all service startups including cert-manager — write await exceeded 800ms for 6 consecutive minutes", - "evidence": "/tmp/microshift-ci-claude-workdir.260601/graphs/123456/3_disk_io.png:1", - "quote": ""} + "evidence": "/tmp/microshift-ci-claude-workdir.260601/graphs/123456/io.json:42", + "quote": "\"await\": [823.5,"} ], "confidence": "medium", "analysis_gaps": [], @@ -97,7 +97,7 @@ Each entry in the output array has exactly these fields: - `release`: extract from job_name (e.g. `4.22` from `release-4.22`), default `main` - `remediation`: suggested fix (~120 chars). Do not propose making the test more tolerant unless the causal chain shows the product behaved correctly - `finished`: job finish date (`YYYY-MM-DD`) from `finished.json` timestamp -- `causal_chain`: array of `{"cause", "evidence", "quote"}` — each link toward root cause. `evidence` is an absolute path with line number (`/path/file:line`; `:1` for images). `quote` is a short verbatim excerpt (empty for images). Re-read every cited `file:line` before finalizing. Aim for 2-4 links. +- `causal_chain`: array of `{"cause", "evidence", "quote"}` — each link toward root cause. `evidence` is an absolute path with line number (`/path/file:line`; `:1` for binary files). `quote` is a short verbatim excerpt (empty for binary files). Re-read every cited `file:line` before finalizing. Aim for 2-4 links. - `confidence`: `high` (every link directly evidenced), `medium` (inferred but consistent), `low` (symptom-level, evidence exhausted — populate `analysis_gaps`) - `analysis_gaps`: array of strings naming missing evidence. Empty when nothing was skipped. - `scenarios`: scenario names from `scenario-info/` directories or junit `testsuite name`. Empty array for non-scenario failures. @@ -136,7 +136,7 @@ One line, ~80 chars. Focus on the mechanism. Use stable terms — the same under Downstream automation uses confidence to decide whether to act — do not inflate it. -- `high`: every causal-chain link is directly evidenced by a quoted artifact line or graph +- `high`: every causal-chain link is directly evidenced by a quoted artifact line or metric data point - `medium`: the mechanism is inferred but consistent with all available evidence - `low`: symptom-level only — populate `analysis_gaps` diff --git a/plugins/microshift-ci/scripts/pcp-graphs/README.md b/plugins/microshift-ci/scripts/pcp-graphs/README.md index 6854dbf6..7918df74 100644 --- a/plugins/microshift-ci/scripts/pcp-graphs/README.md +++ b/plugins/microshift-ci/scripts/pcp-graphs/README.md @@ -1,8 +1,9 @@ -# PCP Performance Graphs for MicroShift CI +# PCP Performance Metrics for MicroShift CI -Generate performance graphs from PCP (Performance Co-Pilot) archives +Extract performance metrics from PCP (Performance Co-Pilot) archives collected during MicroShift CI job runs. Produces CPU, memory, disk I/O, -and disk usage charts that are embedded in the ci-doctor HTML report. +and disk usage JSON files that are embedded as interactive Chart.js charts +in the ci-doctor HTML report. ## Background @@ -47,13 +48,12 @@ The directory contains files like `yyyymmdd.hh.mm.{0,index,meta}` and a ## Prerequisites - `pcp-export-pcp2json` package (provides the `pcp2json` command) -- Python 3 with `matplotlib` +- Python 3 Install: ```bash sudo dnf install -y pcp-export-pcp2json -pip install matplotlib ``` ## Usage @@ -64,42 +64,40 @@ Graphs are generated automatically by `doctor.sh graphs`: bash doctor.sh graphs --workdir /tmp/microshift-ci-claude-workdir.YYMMDD ``` -This finds all PCP archives in downloaded artifacts and produces PNG -graphs at `${WORKDIR}/graphs//`. The `finalize` step then -embeds them as base64 in the HTML report with a tabbed UI per job. +This finds all PCP archives in downloaded artifacts and produces JSON +metric files at `${WORKDIR}/graphs//`. The `finalize` step +embeds the JSON data in the HTML report as interactive Chart.js charts. ## Output | File | Description | |---|---| -| `1_cpu_usage.png` | CPU usage stacked area: User (blue), I/O Wait (orange), System (red) | -| `2_mem_usage.png` | Memory usage stacked area: Used (red), Cached (orange), Total (dashed) | -| `3_disk_io.png` | Disk I/O chart: Read OPS (blue), Write OPS (red), Await (green dashed) | -| `4_disk_usage.png` | Disk usage by partition: fill % per mount point | - -Numeric prefixes control tab display order in the HTML report. +| `cpu.json` | CPU usage: user%, sys%, iowait%, idle% at 15s intervals | +| `mem.json` | Memory usage: used_gb, cached_gb, free_gb, total_gb at 15s intervals | +| `io.json` | Disk I/O: read/write ops/s, iops, await (ms), queue depth at 15s intervals | +| `disk.json` | Disk usage per partition: used_pct%, used_gb at 15s intervals | ## Files | File | Purpose | |---|---| -| `generate-graphs.sh` | Orchestrator: finds PCP archives, runs extraction and plotting in parallel | +| `generate-graphs.sh` | Orchestrator: finds PCP archives, runs extraction in parallel | | `extract_cpu.sh` | Runs `pcp2json` for CPU metrics, pipes through `parse_cpu.py` | | `parse_cpu.py` | Parses pcp2json CPU output (user, sys, iowait, idle), normalizes to percentages | -| `plot_cpu.py` | Generates CPU usage PNG with stacked area chart and peak table | | `extract_mem.sh` | Runs `pcp2json` for memory metrics, pipes through `parse_mem.py` | | `parse_mem.py` | Parses pcp2json memory output (used, free, cached, physmem), converts to GB | -| `plot_mem.py` | Generates memory usage PNG with stacked area chart | | `extract_io.sh` | Runs `pcp2json` for disk metrics, pipes through `parse_pcp.py` | | `parse_pcp.py` | Parses pcp2json disk output, aggregates per-device (sum read/write, max await) | -| `plot_io.py` | Generates disk I/O PNG with dual Y-axes (OPS + await) | | `extract_disk_usage.sh` | Runs `pcp2json` for filesystem metrics, pipes through `parse_disk_usage.py` | | `parse_disk_usage.py` | Parses pcp2json filesys output, tracks all partitions as usage percentages | -| `plot_disk_usage.py` | Generates per-partition disk usage PNG with mount points in legend | +| `pcp-charts.js` | Shared Chart.js rendering functions for CPU/mem/io/disk charts | +| `create-pcp-dashboard.py` | Standalone interactive HTML dashboard from PCP metrics | +| `generate-dashboard.sh` | Per-scenario dashboard generator (handles per-VM PCP tarballs) | +| `plot_*.py` | Legacy matplotlib plotters (no longer called by the pipeline) | -## Adding a New Graph Type +## Adding a New Metric Type 1. Create `extract_.sh` and `parse_.py` (follow existing patterns) -2. Create `plot_.py` -3. Add a block to `generate-graphs.sh` with the next numeric prefix (e.g. `5_`) -4. No changes needed in `create-report.py` — it auto-discovers all `*.png` files +2. Add a block to `generate-graphs.sh` to extract the new metric +3. Add a rendering function to `pcp-charts.js` +4. Update `create-report.py` to load the new JSON file in `_METRIC_FILES` diff --git a/plugins/microshift-ci/skills/doctor/SKILL.md b/plugins/microshift-ci/skills/doctor/SKILL.md index 52c5d5c8..eb470c80 100644 --- a/plugins/microshift-ci/skills/doctor/SKILL.md +++ b/plugins/microshift-ci/skills/doctor/SKILL.md @@ -78,12 +78,12 @@ Compute once at the start by running `date +%y%m%d` and substituting into the pa bash plugins/microshift-ci/scripts/doctor.sh graphs --component microshift --workdir ``` -2. The script finds PCP archives in downloaded artifacts and generates PNG graphs at `/graphs//`: - - `1_cpu_usage.png` — CPU usage (user, system, I/O wait) - - `2_mem_usage.png` — Memory usage (used, cached) - - `3_disk_io.png` — Disk I/O (read/write OPS, await) - - `4_disk_usage.png` — Disk usage by partition (% fill) -3. If prerequisites are missing (`pcp2json`, `matplotlib`), the script errors and stops. +2. The script finds PCP archives in downloaded artifacts and generates JSON metric files at `/graphs//`: + - `cpu.json` — CPU usage (user, system, I/O wait, idle) + - `mem.json` — Memory usage (used, cached, free, total) + - `io.json` — Disk I/O (read/write ops, await, queue depth) + - `disk.json` — Disk usage by partition (% fill, GB used) +3. If prerequisites are missing (`pcp2json`), the script errors and stops. ### Step 2: Analyze Each Job Using microshift-ci:prow-job-analyzer Agent @@ -217,8 +217,7 @@ HTML report generated: /report-microshift-ci-doctor.html - MCP Jira server must be configured (for bug correlation) - Internet access to fetch job data from Prow/GCS - Bash shell, Python 3 -- `pcp-export-pcp2json` — for PCP graph generation -- `matplotlib` Python package — for PCP graph plotting +- `pcp-export-pcp2json` — for PCP metric extraction ## Related Skills