diff --git a/AGENTS.md b/AGENTS.md index 89dde36..463fd50 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -21,7 +21,8 @@ This project provides AI-driven tools for end-to-end feature development in Open | `/oape:e2e-generate ` | Generate e2e test artifacts from git diff against base branch | | `/oape:predict-regressions ` | Predict API regressions and breaking changes from git diff | | `/oape:review [base_ref]` | Production-grade code review against Jira requirements | -| `/oape:implement-review-fixes ` | Automatically apply fixes from a review report | +| `/oape:implement-review-fixes ` | Automatically apply fixes from a review report | +| `/oape:pr-agent [--dry-run] [--monitor-only]` | Monitor PR CI status, classify failures, generate report | ### Input Sources for api-generate and api-implement @@ -102,7 +103,7 @@ When using a design document (gist), it should contain structured implementation ## Supported Operator Repositories -The allowed repositories and their base branches are defined in [`team-repos.csv`](config/team-repos.csv). DO NOT raise PRs on any repos beyond that list. Always read `team-repos.csv` to determine the correct repo URL and base branch before cloning or creating branches. +The allowed repositories and their base branches are defined in [`team-repos.csv`](deploy/config/team-repos.csv). DO NOT raise PRs on any repos beyond that list. Always read `team-repos.csv` to determine the correct repo URL and base branch before cloning or creating branches. --- @@ -119,7 +120,19 @@ The commands automatically detect which framework the repository uses: ## Project Structure - +``` +scripts/ci-monitor/ CI monitoring pipeline (monitor.sh, dispatch.sh) +scripts/pr-agent/ PR lifecycle agent (entrypoint, auto-fix, safety, test-dry-run) +plugins/oape/commands/ Claude Code command definitions (/oape:* commands) +plugins/oape/skills/ Claude Code skill definitions (injected into agent prompts) +config/ Tool configuration (config.json) +images/ Container image Dockerfiles (ci-monitor, agent-worker, go-server, gh-token-minter) +docs/ Architecture docs, Prow config templates +deploy/ Kubernetes deployment manifests +go-server/ Go HTTP server for agent job execution +agent/ Python agent wrapper +gh-token-minter/ GitHub App token generation utility +``` --- diff --git a/OAPE-710-subtasks.md b/OAPE-710-subtasks.md new file mode 100644 index 0000000..71dbbb0 --- /dev/null +++ b/OAPE-710-subtasks.md @@ -0,0 +1,1784 @@ +# PR Lifecycle Agent — Subtasks + +**Parent Ticket:** [OAPE-710](https://redhat.atlassian.net/browse/OAPE-710) +**Type:** Story +**Status:** To Do +**Assignee:** Neha Kumari + +> **Phase 1 Scope Change (2026-06-10, updated 2026-06-18 for Prow migration):** Phase 1 is a **Prow presubmit CI monitor** (`oape-ci-monitor`) configured centrally in `openshift/release` for each target repo. The presubmit runs alongside other CI jobs, polls until all checks reach a terminal state, then classifies failures, queries Sippy for flake history, and posts a structured report as a PR comment. Phase 1 is **report-only**: no auto-fix, no review comment handling, no Claude dependency. The report includes a machine-readable JSON output with suggested trigger actions for future phases. The job builds a `ci-monitor-agent` container image inline via ci-operator's `dockerfile_literal` and mounts Prow-managed secrets for GCP and GitHub App credentials. Configuration template: `docs/prow-ci-operator-config.yaml`. + +## Motivation & Goals + +### The Problem + +OAPE automates the code-generation half of the feature development lifecycle — from Enhancement Proposal to API types, tests, and controller implementation. But once a PR is opened, the journey from **"PR created" to "PR merged" is entirely manual**. Developers are left to monitor CI checks, dig through Prow and GitHub Actions logs, fix trivial lint and formatting failures, parse noisy bot comments, track reviewer feedback, and push fixes — all by hand. + +This manual loop is both time-consuming and mechanical. Each CI round-trip (fail → read logs → fix → push → wait for CI) takes 15–30 minutes. Trivial failures — formatting, import ordering, missing generated files — account for a large share of CI failures on OAPE-generated code, yet each one requires the same checkout-fix-verify-push cycle. Across multiple PRs and repos, this adds up to hours of wasted developer time per week. + +The PR agent closes this gap by automating the post-PR lifecycle as a **Prow presubmit CI job**: CI monitoring, failure triage, trivial auto-fixing, review comment addressing, and status reporting — all running autonomously alongside other CI checks or on-demand via `/test oape-ci-monitor`. + +### Why Prow Presubmit (Not K8s Jobs or GitHub Actions) + +The existing OAPE execution model uses K8s Jobs via the go-server for code generation workloads. The PR agent uses Prow presubmit jobs instead because: + +- **Native OpenShift CI integration**: Prow presubmits run alongside existing CI jobs in the same infrastructure — no separate runner fleet or workflow files per repo +- **Centralized configuration**: Job definitions live in `openshift/release` (`ci-operator/config/`), not scattered as `.github/workflows/*.yml` across target repos +- **Prow secret management**: Secrets (GCP ADC, GitHub App credentials) are mounted from the `test-credentials` namespace — no per-repo GitHub Actions secrets to configure +- **ci-operator image build**: The `ci-monitor-agent` container is built inline via `dockerfile_literal`, ensuring consistent dependencies across all target repos +- **ChatOps trigger**: Manually triggerable via `/test oape-ci-monitor` on any PR — no `workflow_dispatch` UI needed +- The go-server/K8s Job model is optimized for long-running code generation workloads that need specific tools and cluster access — the PR agent's presubmit-driven CI monitoring pattern integrates naturally with OpenShift CI + +Human Review Required + +> **AI-generated code must not be relied upon without human review.** All fixes pushed by these jobs must go through the standard GitHub PR review process. Repository OWNERS are responsible for reviewing and approving all changes. + +### Before & After Workflow + +``` +BEFORE (manual): + Developer opens PR + → Polls `gh pr checks` repeatedly + → Reads CI logs (Prow/GCS, GitHub Actions) + → Identifies failure: "oh, it's just goimports" + → Checks out branch, runs goimports, verifies build + → Commits, pushes, waits 20 min for CI + → Repeats for next failure + → Reads 15 review comments, 10 are bots + → Identifies 2 actionable items + → Fixes, pushes again + Total: 2–4 hours of mechanical work + +AFTER (with Prow presubmit ci-monitor): + Developer opens PR + → CI runs (Prow presubmits fire, including oape-ci-monitor) + → oape-ci-monitor polls until all other checks reach terminal state + → Agent classifies failures deterministically (regex + Sippy) + → Agent posts structured CI analysis report as PR comment + → (Phase 2+) Agent auto-fixes trivial CI failures, addresses reviews + → Developer sees failure categories, flake rates, and recommended actions + → Developer focuses on actionable failures only + Total: Developer spends ~10 min triaging CI results instead of reading raw logs +``` + +### Pain Points & Solutions + + +| Pain Point | Impact | PR Agent Solution | +| ------------------------------------------ | ------------------------------ | ---------------------------------------------------------------------------------------------- | +| Repeated manual CI polling | Context switching, wasted time | Prow presubmit runs alongside CI and reports once all checks complete — no manual polling needed | +| Fixing lint/format/generated-file failures | 15–30 min per round-trip | Auto-fix engine applies `go fmt`, `goimports`, `make generate` | +| Parsing noisy bot comments | Signal buried in noise | Categorizes comments, filters bots, surfaces actionable items only | +| Waiting between CI re-runs | Hours of idle-but-blocked time | Agent pushes fixes immediately, compresses feedback loop | +| Uncertainty about PR readiness | "What still needs to happen?" | Structured status report posted as PR comment | + + +### Capabilities at a Glance + + +| Capability | Description | +| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| CI Monitoring | Fetches all CI checks (GitHub Actions + Prow), categorizes status as passed/failed/pending | +| Failure Analysis | Classifies failures as trivial (auto-fixable) vs. non-trivial (requires human attention) via Claude Code | +| Auto-Fix Engine | Runs the correct fix command, verifies compilation, commits and pushes | +| Review Comment Handling | Analyzes unresolved review threads, addresses actionable feedback via Claude Code | +| Trigger Modes | Prow presubmit (automatic on every PR push in configured repos, also triggerable via `/test oape-ci-monitor`) | +| Safety Guardrails | File blocklists, commit limits, audit logging, dry-run mode | +| Status Reporting | Markdown report posted as PR comment; build logs available in Prow GCS artifacts | + + +### Prior Art: HyperShift AI-Assisted CI Jobs + +This feature follows the pattern established by [HyperShift's AI-assisted CI jobs](https://hypershift.pages.dev/how-to/ci/ai-assisted-ci-jobs/), which use GitHub Actions workflows powered by Claude Code to automate Jira issue resolution, PR review comment handling, and dependabot triage. The PR Lifecycle Agent adapts this pattern for code-generation workflows, where failure patterns are more predictable (missing schemes, RBAC consistency, generated file sync). + +Key design parallels with HyperShift: + + +| Aspect | HyperShift | OAPE PR Agent | +| ----------------- | --------------------------------------- | -------------------------------------------------------------------------------------------------- | +| CI platform | GitHub Actions | Prow presubmit (ci-operator) | +| AI engine | Claude Code CLI via Vertex AI | Claude Code CLI via Vertex AI | +| Periodic scanner | `periodic-review-agent` (every 3h) | N/A (presubmit-driven, no periodic sweep in Phase 1) | +| On-demand trigger | `/test address-review-comments` | `/test oape-ci-monitor` (Prow chatops) | +| PR scope | `app/hypershift-jira-solve-ci` PRs only | All open PRs in allowed repos (`team-repos.csv`) | +| Max items per run | 10 PRs (review agent) | 4 PRs (configurable via `PR_AGENT_MAX_PRS`) | +| Max budget per PR | $5.00 per PR | $5.00 per PR (configurable via `MAX_BUDGET_PER_PR`, passed to `--max-budget-usd`) | +| Safety | Draft PRs only, human review required | File blocklists, commit limits, audit log | + + +### Expected Outcome + +Once the PR Lifecycle Agent is complete, all open PRs in allowed repos (`team-repos.csv`) will be automatically monitored via a Prow presubmit job (`oape-ci-monitor`) configured centrally in `openshift/release`. The presubmit runs alongside other CI checks and reports once all are terminal. The agent will classify CI failures, post structured analysis reports, and (in later phases) fix trivial CI failures and address review comments — all without developer intervention. Developers can also trigger the agent on-demand via `/test oape-ci-monitor`. The measurable goal is to **eliminate manual trivial-fix round-trips** and reduce time from PR-opened to CI-green from hours to minutes for the common case. + +--- + +## Completion Status + +| # | Subtask | Phase | Status | Notes | +|---|---------|-------|--------|-------| +| 0 | Prow presubmit infrastructure | 1 | **Done** | `docs/prow-ci-operator-config.yaml`, `images/ci-monitor.Dockerfile`, release PR #80727 (rehearsal passed) | +| 1 | Entrypoint + PR discovery | 1 | **Done** | `scripts/pr-agent/entrypoint.sh` — periodic/on-demand modes, state persistence, per-PR timeout | +| 2 | CI check monitoring | 1 | **Done** | Dual implementation: lightweight in entrypoint.sh, comprehensive in `scripts/ci-monitor/monitor.sh` | +| 3 | Failure log analysis | 1 | **Partial** | Deterministic regex classification done. Claude fallback for `unknown` deferred to Phase 2 | +| 4 | Trivial auto-fix engine | 2 | **Partial** | `trivial-format`, `trivial-generated-files`, and `lint-failure` (treated as format) implemented in auto-fix.sh. Fine-grained `trivial-import`/`trivial-lint` classification deferred | +| 5 | Review comment handler | 2 | **In progress** | PR #63 (`oape-review-handler`) implements `review-handler.sh`, `address-review-comments` skill, `pr-agent-safety` skill, and Prow config. Open, not yet merged. | +| 6 | Pipeline wiring | 1 | **Done** | `process_pr()` orchestrates all phases; `dispatch.sh` routes actions (Phase 1 = log-only) | +| 7 | Safety guardrails | 1 | **Done** | `scripts/pr-agent/safety.sh` — blocklist, commit limits, audit log, retry helpers | +| 8 | Status reporting | 1 | **Done** | Report generation + idempotent PR comment posting in entrypoint.sh | +| 9 | Testing & validation | 1 | **Done** | `scripts/pr-agent/test-dry-run.sh` — shellcheck + dry-run integration + output verification | +| 10 | `/oape:pr-agent` command | 1 | **Done** | `plugins/oape/commands/pr-agent.md` + AGENTS.md command table | + +**Phase 1 (report-only):** Complete — core infrastructure done (subtasks 0-2, 6-10). Subtasks 3, 4 partial. +**Phase 2 (auto-fix + review):** Subtask 4 (expand) and 3 (Claude fallback) remain. Subtask 5 in progress via PR #63. + +--- + +## Subtask Overview + +This document breaks the PR Lifecycle Agent into 10 implementable subtasks. Each subtask is self-contained with a clear definition, acceptance criteria, dependencies, and implementation hints. The architecture follows a **hybrid model**: deterministic bash for mechanical orchestration (PR discovery, CI polling, tool setup, safety guardrails) and Claude Code CLI for intelligent analysis (failure classification, review comment handling, complex code fixes). + +### Dependency Graph + +``` +Subtask 0 (Prow Presubmit Infrastructure + ci-operator Config) +├── Subtask 1 (Entrypoint + PR Discovery + State Tracking) +│ ├── Subtask 2 (CI Monitoring) +│ │ └── Subtask 3 (Log Analysis + Deterministic Classification) +│ ├── Subtask 7 (Safety Guardrails) ← no dependencies (standalone utility) +│ │ └── Subtask 4 (Auto-Fix) ← depends on 3 + 7 +│ ├── Subtask 5 (Review Comments) +│ ├── Subtask 6 (Wire Processing Pipeline) ← depends on 2–5, 7–8 +│ │ Subtask 8 (Status Reporting) ← depends on 2–7 +│ │ Subtask 9 (Testing) ← depends on all above +│ └── Subtask 10 (/oape:pr-agent Command) ← depends on 1–8 +``` + +> **Note:** Subtask 7 (Safety Guardrails) is a standalone utility module with no dependencies +> on other subtasks. It provides blocklist, commit limit, audit log, and retry helper functions +> consumed by Subtasks 1, 2, 4, 5, and 6. Pipeline scripts (ci-monitor, auto-fix, review-handler, +> report) are standalone executables communicating via JSON files in `$RUNNER_TEMP`. +> `safety.sh` is a **sourced utility library** (`source scripts/pr-agent/safety.sh`) providing +> shared functions to all scripts that need them. + +### Responsibility Split + + +| Layer | Responsibility | Implementation | +| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | +| **Prow ci-operator config** | Presubmit job definition, container image build (`ci-monitor-agent`), secret mounts, resource requests, timeout configuration | `docs/prow-ci-operator-config.yaml` (template); actual config in `openshift/release` | +| **Bash scripts** | PR discovery, CI status polling, deterministic failure classification, safety guardrails, audit logging, reporting. Pipeline scripts are **standalone executables** communicating via JSON files in `$RUNNER_TEMP`. `safety.sh` is a **sourced utility library** providing shared functions (blocklist, audit log, retry helper, commit counter). | `scripts/pr-agent/*.sh` | +| **Claude Code CLI** | Fallback failure classification (for `unknown` categories), review comment analysis/response, complex code fixes. Skills included via `cat` in the prompt. | `plugins/oape/skills/*.md` content injected via `claude --print -p "$(cat SKILL.md) ..."` | + + +### Two-Layer Classification Taxonomy + +The system uses two classification layers that serve different purposes: + +| Layer | Script | Categories | Purpose | +| ----- | ------ | ---------- | ------- | +| **Job-level (triage)** | `scripts/ci-monitor/monitor.sh` | `install-failure`, `build-failure`, `lint-failure`, `test-failure`, `infra-flake`, `unknown` | Classifies CI job failures for reporting and triage. Maps to high-level actions: `retest`, `auto-fix-lint`, `investigate`. | +| **Fix-level (actionable)** | `scripts/pr-agent/entrypoint.sh` | `trivial-format`, `trivial-import`, `trivial-lint`, `trivial-generated-files`, `build-error`, `test-failure`, `infra-flake`, `unknown` | Classifies failures by the specific fix command needed: `go fmt`, `goimports`, `make generate`, etc. | + +`dispatch.sh` bridges the two layers: it reads `monitor.sh`'s job-level classification (e.g., `lint-failure` → `auto-fix-lint` action) and invokes the appropriate `pr-agent/` script, which performs the finer-grained fix-level classification to determine the exact fix command. + + +--- + +## Subtask 0: Prow presubmit infrastructure and ci-operator configuration + +### Description + +Establish the Prow presubmit job infrastructure that all subsequent subtasks build upon. This includes the ci-operator config snippet that defines the `ci-monitor-agent` container image (built inline via `dockerfile_literal`) and the `oape-ci-monitor` presubmit test step. The config is added to each target repo's ci-operator config in `openshift/release`. + +> **Phase 1 implementation:** A single `oape-ci-monitor` presubmit job runs as `always_run: true, optional: true` alongside other CI jobs. It polls until all other checks reach a terminal state, then runs `monitor.sh` and `dispatch.sh`. Manually triggerable via `/test oape-ci-monitor`. + +### Acceptance Criteria + +1. A reference ci-operator config exists at `docs/prow-ci-operator-config.yaml` with three snippets: + - Inline `ci-monitor-agent` image build under `images.items[]` + - Promotion exclusion under `promotion.to[].excluded_images` + - Presubmit test definition under `tests[]` +2. The `ci-monitor-agent` image is built from `registry.access.redhat.com/ubi9/go-toolset` and installs: `git`, `make`, `jq`, `gh` CLI. It clones `oape-ai-e2e` at build time and copies scripts/plugins/config into the image. It installs `goimports` and `golangci-lint`. +3. The presubmit job is defined as `always_run: true, optional: true` with job name `oape-ci-monitor`. +4. Authentication is configured with a fallback strategy: + - **Primary:** GitHub App installation token generated inline via JWT signing from the PEM key mounted at `/var/run/github-app/private-key.pem` (secret: `openshift-app-platform-shift-github-bot` in `test-credentials` namespace). Required for Phase 2+ auto-fix pushes that must trigger downstream CI. + - **Fallback:** If the GitHub App is not installed on the target repo, falls back to `GITHUB_TOKEN` (Prow-provided). Sufficient for Phase 1 (read + comment only). Logs a warning with instructions to install the App for Phase 2+. + - Claude API access via GCP Application Default Credentials mounted at `/var/run/gcloud-adc/application_default_credentials.json` (secret: `oap-lts-claude-gcp-vertex-sa` in `test-credentials` namespace). +5. Job timeout is set to `2h30m0s`. Resource requests: 1 CPU, 500Mi memory. +6. The test step invokes `/app/scripts/ci-monitor/monitor.sh` followed by `/app/scripts/ci-monitor/dispatch.sh`. +7. Manually triggerable via `/test oape-ci-monitor` on any PR in a configured target repo. +8. **Rehearsal detection:** When the job runs as a Prow rehearsal (i.e., `REPO_NAME=release` and `REPO_OWNER=openshift`), it detects the `openshift/release` context and switches to a real open PR on the target repo (e.g., `openshift/must-gather-operator`). The rehearsal runs the full pipeline — including posting the analysis comment on the target PR — to validate the end-to-end flow without requiring the release PR to be merged first. The first open PR on the target repo is selected via the GitHub API. + +### Dependencies + +None — this is the foundation subtask. + +### Implementation Hints + +- **ci-operator config template** (`docs/prow-ci-operator-config.yaml`): + The config contains three snippets to add to the target repo's ci-operator config in `openshift/release` at `ci-operator/config/REPO_ORG/REPO_NAME/REPO_ORG-REPO_NAME-BRANCH.yaml`: + + 1. **Inline image build** — builds the `ci-monitor-agent` container: + ```yaml + images: + items: + - dockerfile_literal: |- + FROM registry.access.redhat.com/ubi9/go-toolset + USER 0 + RUN dnf install -y git make jq && \ + dnf install -y 'dnf-command(config-manager)' && \ + dnf config-manager --add-repo https://cli.github.com/packages/rpm/gh-cli.repo && \ + dnf install -y gh && \ + dnf clean all + WORKDIR /app + RUN git clone --depth 1 -b main https://github.com/openshift-eng/oape-ai-e2e.git /tmp/oape && \ + cp -r /tmp/oape/scripts /app/scripts && \ + cp -r /tmp/oape/plugins /plugins && \ + mkdir -p /config && cp -r /tmp/oape/deploy/config/* /config/ && \ + rm -rf /tmp/oape + RUN go install golang.org/x/tools/cmd/goimports@latest && \ + curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b /usr/local/bin + RUN git config --global user.name "openshift-app-platform-shift-bot" && \ + git config --global user.email "267347085+openshift-app-platform-shift-bot@users.noreply.github.com" + RUN chmod -R g=u /opt/app-root/src + USER 1001 + to: ci-monitor-agent + ``` + + 2. **Presubmit test step** — runs the CI monitor: + ```yaml + - always_run: true + as: oape-ci-monitor + optional: true + steps: + test: + - as: monitor + commands: | + set -euo pipefail + + echo "[setup] Starting oape-ci-monitor for ${REPO_OWNER}/${REPO_NAME} PR#${PULL_NUMBER}" + + # --- Rehearsal detection --- + # Prow rehearsal runs against openshift/release, not the target repo. + # Switch to a real target-repo PR to validate the full pipeline. + if [[ "${REPO_NAME}" == "release" && "${REPO_OWNER}" == "openshift" ]]; then + echo "[setup] Detected openshift/release context — switching to test target" + export REPO_OWNER="REPO_ORG" + export REPO_NAME="REPO_NAME" + TEST_PR=$(curl -s "https://api.github.com/repos/${REPO_OWNER}/${REPO_NAME}/pulls?state=open&per_page=1" \ + | python3 -c "import sys,json; data=json.load(sys.stdin); print(data[0]['number'] if data else '')" 2>/dev/null || echo "") + if [[ -z "$TEST_PR" ]]; then + echo "[setup] No open PRs found on ${REPO_OWNER}/${REPO_NAME} — skipping" + exit 0 + fi + export PULL_NUMBER="$TEST_PR" + export PR_URL="https://github.com/${REPO_OWNER}/${REPO_NAME}/pull/${PULL_NUMBER}" + echo "[setup] Testing against ${REPO_OWNER}/${REPO_NAME}#${PULL_NUMBER}" + fi + + # --- GitHub auth: try App token, fall back to GITHUB_TOKEN --- + # App token is preferred (required for Phase 2+ pushes that trigger CI). + # For Phase 1 (report-only), GITHUB_TOKEN is sufficient for read + comment. + USE_APP_TOKEN="false" + if [[ -f /var/run/github-app/app-id && -f /var/run/github-app/private-key.pem ]]; then + echo "[auth] Attempting GitHub App token..." + APP_ID=$(cat /var/run/github-app/app-id) + PEM_PATH="/var/run/github-app/private-key.pem" + HEADER=$(printf '{"alg":"RS256","typ":"JWT"}' | openssl base64 -e -A | tr '+/' '-_' | tr -d '=') + NOW=$(date +%s); EXP=$((NOW + 300)) + PAYLOAD=$(printf '{"iat":%d,"exp":%d,"iss":"%s"}' "$NOW" "$EXP" "$APP_ID" | openssl base64 -e -A | tr '+/' '-_' | tr -d '=') + SIGNATURE=$(printf '%s' "${HEADER}.${PAYLOAD}" | openssl dgst -sha256 -sign "$PEM_PATH" -binary | openssl base64 -e -A | tr '+/' '-_' | tr -d '=') + JWT="${HEADER}.${PAYLOAD}.${SIGNATURE}" + + INSTALL_RESPONSE=$(curl -s -w "\n%{http_code}" -H "Authorization: Bearer ${JWT}" -H "Accept: application/vnd.github+json" \ + "https://api.github.com/repos/${REPO_OWNER}/${REPO_NAME}/installation") + HTTP_CODE=$(echo "$INSTALL_RESPONSE" | tail -1) + INSTALL_BODY=$(echo "$INSTALL_RESPONSE" | sed '$d') + + if [[ "$HTTP_CODE" -eq 200 ]]; then + INST_ID=$(echo "$INSTALL_BODY" | python3 -c "import sys,json; print(json.load(sys.stdin)['id'])") + TOKEN_RESPONSE=$(curl -s -w "\n%{http_code}" -X POST -H "Authorization: Bearer ${JWT}" -H "Accept: application/vnd.github+json" \ + "https://api.github.com/app/installations/${INST_ID}/access_tokens") + T_CODE=$(echo "$TOKEN_RESPONSE" | tail -1) + T_BODY=$(echo "$TOKEN_RESPONSE" | sed '$d') + if [[ "$T_CODE" -eq 201 ]]; then + export GH_TOKEN=$(echo "$T_BODY" | python3 -c "import sys,json; print(json.load(sys.stdin)['token'])") + USE_APP_TOKEN="true" + echo "[auth] GitHub App token generated successfully" + else + echo "[auth] WARN: App token creation failed (HTTP ${T_CODE}), falling back to GITHUB_TOKEN" + fi + else + echo "[auth] WARN: App not installed on ${REPO_OWNER}/${REPO_NAME} (HTTP ${HTTP_CODE}), falling back to GITHUB_TOKEN" + fi + else + echo "[auth] GitHub App credentials not mounted, using GITHUB_TOKEN" + fi + + if [[ "$USE_APP_TOKEN" != "true" ]]; then + if [[ -z "${GH_TOKEN:-}" && -z "${GITHUB_TOKEN:-}" ]]; then + echo "[auth] ERROR: No GitHub token available (App token failed and GITHUB_TOKEN not set)" >&2 + exit 1 + fi + export GH_TOKEN="${GH_TOKEN:-${GITHUB_TOKEN}}" + echo "[auth] Using GITHUB_TOKEN (Phase 1 report-only — sufficient for read + comment)" + echo "[auth] NOTE: Phase 2+ auto-fix pushes require the GitHub App to be installed on ${REPO_OWNER}/${REPO_NAME}" + fi + + # --- GCP auth for Claude (Vertex AI) --- + export GOOGLE_APPLICATION_CREDENTIALS="/var/run/gcloud-adc/application_default_credentials.json" + export CLAUDE_CODE_USE_VERTEX="1" + export CLOUD_ML_REGION="global" + export ANTHROPIC_VERTEX_PROJECT_ID="itpc-gcp-hcm-pe-eng-claude" + + # --- Run CI monitor --- + export PR_URL="https://github.com/${REPO_OWNER}/${REPO_NAME}/pull/${PULL_NUMBER}" + export SKIP_POLL="false" + export SELF_JOB_NAME="oape-ci-monitor" + export BUILD_ID="${BUILD_ID:-}" + export OAPE_RUN_URL="${BUILD_LOG_URL:-}" + + gh auth setup-git + /app/scripts/ci-monitor/monitor.sh + /app/scripts/ci-monitor/dispatch.sh + credentials: + - mount_path: /var/run/gcloud-adc + name: oap-lts-claude-gcp-vertex-sa + namespace: test-credentials + - mount_path: /var/run/github-app + name: openshift-app-platform-shift-github-bot + namespace: test-credentials + from: ci-monitor-agent + resources: + requests: + cpu: "1" + memory: 500Mi + timeout: 2h30m0s + ``` + +- **GitHub App token** is preferred over `GITHUB_TOKEN` because pushes made with `GITHUB_TOKEN` do not trigger downstream CI runs (GitHub's anti-recursion rule). The Prow job attempts to generate the App token inline via JWT signing from the mounted PEM key. If the App is not installed on the target repo, it falls back to `GITHUB_TOKEN` which is sufficient for Phase 1 (read + comment only). Phase 2+ auto-fix pushes require the App to be installed. +- **Rollout**: To add the CI monitor to a target repo, copy the three snippets from `docs/prow-ci-operator-config.yaml` into the target repo's ci-operator config in `openshift/release` and submit a PR. The rehearsal detection block allows validating the full pipeline (including comment posting on the target repo) via `/pj-rehearse` before merging the release PR. + +### Files + + +| File | Action | +| ----------------------------------- | -------------------------------------------------------------------- | +| `docs/prow-ci-operator-config.yaml` | Create (reference ci-operator config for target repos) | + + +--- + +## Subtask 1: Create entrypoint script with PR discovery and prechecks + +### Description + +Create the main bash entrypoint script that orchestrates the PR agent workflow. This script handles two execution modes: **periodic** (discovers all open PRs across allowed repos and processes each) and **on-demand** (processes a single specified PR). It includes argument parsing, prechecks, and the top-level processing loop that invokes downstream capabilities (CI monitoring, auto-fix, review handling). + +> **Phase 1 implementation:** The entrypoint supports a `--monitor-only` flag that skips the auto-fix phase entirely, running only CI monitoring, deterministic classification, and status reporting. Phase 1 uses on-demand + `--monitor-only` mode exclusively. The periodic sweep and auto-fix code paths exist but are not exercised until Phase 2. + +### Acceptance Criteria + +1. Entrypoint script accepts `--mode` flag with values `periodic` or `on-demand`. +2. In `periodic` mode: + - Queries GitHub for all open PRs across repos listed in `deploy/config/team-repos.csv`. + - Processes up to `PR_AGENT_MAX_PRS` (default 4) PRs per run. Kept low to ensure the run completes within the Prow job timeout (`2h30m0s`). + - Adds a 60-second delay between processing each PR (rate limiting). +3. In `on-demand` mode: + - Accepts `--pr-url ` argument. + - Parses PR URL in both formats: full URL (`https://github.com/org/repo/pull/123`) and shorthand (`org/repo#123`). Extracts owner, repo name, and PR number. + - Validates the PR exists and is in `open` state. + - Validates the PR targets a repo listed in `deploy/config/team-repos.csv`. Rejects PRs from repos not in the allowlist. +4. In `periodic` mode, filters out PRs with the `pr-agent:skip` label. Developers can add this label to exclude specific PRs from automated processing. +5. Before processing each PR, checks for merge conflicts via `gh pr view --json mergeable -q .mergeable`. If `CONFLICTING`, skips CI analysis and auto-fix phases, proceeding directly to the status report with "merge conflict" as the primary finding. +6. Prechecks all pass before any work begins: + - `gh auth status` confirms GitHub CLI authentication. + - Claude Code CLI is available (`claude --version`). + - Required environment variables are set (`GH_TOKEN`, `CLAUDE_CODE_USE_VERTEX`). +7. Fails immediately with a clear, prefixed error message (e.g., `PRECHECK FAILED: PR #123 is not open`) when any precheck fails. +8. Emits structured log lines to stdout for each PR processed: `[PR #N] owner/repo#123 — processing started`. +9. Stays within Prow job timeout: The Prow presubmit timeout is `2h30m0s`. The GitHub App installation token is generated inline at job start and is valid for 1 hour, which is sufficient for single-PR presubmit processing. The periodic run limits `PR_AGENT_MAX_PRS` to 4 (default) to ensure processing completes within this window. +10. Maintains lightweight state persistence to avoid re-processing: tracks which CI jobs have been analyzed and which review comments have been addressed. State is persisted **across job runs** by embedding a hidden state block in the PR report comment: ``. The state schema includes `analyzed` (array of `name:url` job keys — URL changes after `/retest`, ensuring re-runs get fresh analysis), `addressed` (array of comment IDs), and `last_run` (ISO timestamp). On each run, the agent reads the existing report comment, parses the embedded state, and skips already-processed jobs and comments. Within a run, an in-memory copy in `$RUNNER_TEMP/pr-agent-state---.json` prevents duplicate work across multiple PRs. +11. Wraps `gh` API calls in a retry helper function with exponential backoff (3 retries at 5s/15s/45s intervals) for resilience against transient GitHub API failures. + +### Dependencies + +Subtask 0 (Prow presubmit infrastructure must exist). + +### Implementation Hints + +- **PR discovery for periodic mode:** + ```bash + # Query all open PRs across allowed repos + { + read -r # Skip CSV header row + while IFS=, read -r product role repo_url; do + owner_repo=$(echo "$repo_url" | sed 's|https://github.com/||;s|\.git$||') + prs=$(gh pr list --repo "$owner_repo" \ + --state open --json number,url,headRefName,title,labels --limit 20) + # Filter out PRs with pr-agent:skip label + prs=$(echo "$prs" | jq '[.[] | select(.labels | map(.name) | index("pr-agent:skip") | not)]') + # Append to processing list + done + } < deploy/config/team-repos.csv + ``` +- **PR URL parsing:** + ```bash + parse_pr_url() { + local url="$1" + if [[ "$url" =~ https://github.com/([^/]+)/([^/]+)/pull/([0-9]+) ]]; then + OWNER="${BASH_REMATCH[1]}" + REPO="${BASH_REMATCH[2]}" + PR_NUMBER="${BASH_REMATCH[3]}" + elif [[ "$url" =~ ^([^/]+)/([^#]+)#([0-9]+)$ ]]; then + OWNER="${BASH_REMATCH[1]}" + REPO="${BASH_REMATCH[2]}" + PR_NUMBER="${BASH_REMATCH[3]}" + else + echo "PRECHECK FAILED: Invalid PR URL format: $url" >&2 + return 1 + fi + } + ``` +- **PR validation:** + ```bash + pr_state=$(gh pr view "$PR_URL" --json state -q .state) + if [[ "$pr_state" != "OPEN" ]]; then + echo "PRECHECK FAILED: PR $PR_URL is not open (state: $pr_state)" >&2 + return 1 + fi + ``` +- **Merge conflict detection (run before processing):** + ```bash + check_merge_conflicts() { + local pr_url="$1" + local mergeable + mergeable=$(gh_retry gh pr view "$pr_url" --json mergeable -q .mergeable) + if [[ "$mergeable" == "CONFLICTING" ]]; then + echo "[PR] Merge conflict detected — skipping CI analysis and auto-fix" + return 1 + fi + return 0 + } + ``` +- **GitHub API retry helper:** + ```bash + gh_retry() { + local retries=3 delay=5 + for ((i = 1; i <= retries; i++)); do + if "$@"; then return 0; fi + if [[ "$i" -lt "$retries" ]]; then + echo "[retry] Attempt $i/$retries failed, waiting ${delay}s..." >&2 + sleep "$delay" + delay=$((delay * 3)) + fi + done + echo "[retry] All $retries attempts failed for: $*" >&2 + return 1 + } + ``` +- **Processing loop structure:** + ```bash + process_pr() { + local pr_url="$1" + parse_pr_url "$pr_url" + + echo "[PR #${PR_NUMBER}] ${OWNER}/${REPO}#${PR_NUMBER} — processing started" + + # Phase 0: Merge conflict check + if ! check_merge_conflicts "$pr_url"; then + # Skip to status report with merge conflict finding + scripts/pr-agent/report.sh --pr-url "$pr_url" --merge-conflict + return 0 + fi + + # Phase 1: CI Check Monitoring (Subtask 2) + # Phase 2: Failure Analysis + Auto-Fix (Subtasks 3, 4) + # Phase 3: Review Comment Handling (Subtask 5) + # Phase 4: Status Report (Subtask 8) + + echo "[PR #${PR_NUMBER}] ${OWNER}/${REPO}#${PR_NUMBER} — processing complete" + } + ``` +- **Reference:** HyperShift's Jira Agent iterates over issues with a 60-second rate limit between each. The Review Agent iterates over PRs similarly. + +### Files + + +| File | Action | +| -------------------------------- | ------ | +| `scripts/pr-agent/entrypoint.sh` | Create | + + +--- + +## Subtask 2: Implement CI check monitoring and status polling + +### Description + +Add the CI monitoring capability as a standalone executable script. The agent needs to fetch the current state of all CI checks on the PR, categorize the overall status, and extract details about any failures. This is the primary input that drives the analyze-fix cycle. Uses `gh pr checks` which aggregates both GitHub Actions checks (Checks API) and Prow/OpenShift CI status checks (Status API) in a single call, matching the yolo-agent pattern. + +### Acceptance Criteria + +1. Fetches all CI checks for the PR using `gh pr checks` which aggregates both the Checks API (GitHub Actions) and the Status API (Prow/OpenShift CI) in one call. +2. Categorizes overall PR CI status into one of: `all-passed`, `some-failed`, `all-pending`, `mixed-pending`, `no-checks`. +3. For each failed check, extracts: check name, workflow/job name, conclusion (failure/cancelled/timed_out), and the URL to the failed run. +4. Handles `pending` state correctly — reports it as "in progress" rather than treating it as a failure. When all non-pending checks pass, reports status as `mixed-pending`. +5. Outputs structured JSON to a temp file (`$RUNNER_TEMP/ci-status---.json`) for consumption by downstream scripts. +6. Writes a one-line summary to stdout: `[CI] 8/10 passed | 1 failed | 1 pending`. +7. Is a standalone executable script that accepts `--owner`, `--repo`, and `--pr-number` arguments and writes output to `$RUNNER_TEMP`. + +### Dependencies + +Subtask 1 (entrypoint must exist and provide PR context variables). + +### Implementation Hints + +- **CI status fetching via `gh pr checks`** (aggregates both Checks API and Status API): + ```bash + fetch_ci_status() { + local owner="$1" repo="$2" pr_number="$3" + local output_file="${RUNNER_TEMP}/ci-status-${owner}-${repo}-${pr_number}.json" + + # gh pr checks aggregates both GitHub Actions (Checks API) and Prow (Status API) + gh_retry gh pr checks "$pr_number" --repo "${owner}/${repo}" \ + --json name,state,link,bucket \ + > "$output_file" + + # Compute and append summary + local total passed failed pending + total=$(jq 'length' "$output_file") + passed=$(jq '[.[] | select(.bucket == "pass")] | length' "$output_file") + failed=$(jq '[.[] | select(.bucket == "fail")] | length' "$output_file") + pending=$(jq '[.[] | select(.bucket == "pending")] | length' "$output_file") + + echo "[CI] ${passed}/${total} passed | ${failed} failed | ${pending} pending" + } + ``` +- **Status aggregation logic:** + ```bash + aggregate_ci_status() { + local status_file="${RUNNER_TEMP}/ci-status-${1}-${2}-${3}.json" # owner, repo, pr_number + local total passed failed pending + total=$(jq 'length' "$status_file") + passed=$(jq '[.[] | select(.bucket == "pass")] | length' "$status_file") + failed=$(jq '[.[] | select(.bucket == "fail")] | length' "$status_file") + pending=$(jq '[.[] | select(.bucket == "pending")] | length' "$status_file") + + if [[ "$total" -eq 0 ]]; then echo "no-checks" + elif [[ "$failed" -gt 0 ]]; then echo "some-failed" + elif [[ "$pending" -eq "$total" ]]; then echo "all-pending" + elif [[ "$pending" -gt 0 ]]; then echo "mixed-pending" + else echo "all-passed" + fi + } + ``` +- **Reference:** The yolo-agent uses `gh pr checks` with the same `--json name,state,link,bucket` pattern. HyperShift's review agent skips PRs where all checks pass. + +### Files + + +| File | Action | +| -------------------------------- | --------------------------------------------------------- | +| `scripts/pr-agent/ci-monitor.sh` | Create (standalone executable, called by `entrypoint.sh`) | + + +--- + +## Subtask 3: Implement CI failure log analysis and root cause classification + +### Description + +Create the failure log fetching and classification pipeline as a standalone executable script. Classification uses a **two-tier approach**: deterministic regex-based classification first (handles ~80-90% of cases with zero API cost), with Claude Code CLI as a fallback only for failures classified as `unknown`. A Claude Code skill at `plugins/oape/skills/ci-failure-analysis/SKILL.md` serves as the single source of truth for classification taxonomy — its content is included via `cat` in the Claude prompt. + +### Acceptance Criteria + +1. A Claude Code skill exists at `plugins/oape/skills/ci-failure-analysis/SKILL.md` following the project's skill pattern. Its content is included in the Claude prompt via `cat` (not loaded via the plugin system). +2. **Deterministic classification first:** A bash function `classify_failure_deterministic()` uses regex patterns to classify failures without any Claude API call. Covers: `trivial-lint`, `trivial-format`, `trivial-import`, `trivial-generated-files`, `build-error`, `test-failure`, `infra-flake`. +3. **Claude Code as fallback only:** Claude CLI is invoked only for failures classified as `unknown` by the deterministic step. The classification step is read-only — it analyzes logs but never modifies files or runs git write operations. The skill content is included via `cat`: + ```bash + CLASSIFICATION_SCHEMA='{"type":"array","items":{"type":"object","properties":{"category":{"type":"string","enum":["trivial-lint","trivial-format","trivial-import","trivial-generated-files","build-error","test-failure","infra-flake","unknown"]},"confidence":{"type":"string","enum":["high","medium","low"]},"affected_files":{"type":"array","items":{"type":"string"}},"root_cause":{"type":"string"},"suggested_fix":{"type":"string"}},"required":["category","confidence","root_cause"]}}' + + claude --print -p "$(cat plugins/oape/skills/ci-failure-analysis/SKILL.md) + + Analyze the following CI failure log: $(cat "$LOG_FILE")" \ + --allowedTools "Bash(curl*),Read" \ + --json-schema "$CLASSIFICATION_SCHEMA" --max-budget-usd "${MAX_BUDGET_PER_PR:-5.00}" + ``` +4. Classifies each failure into exactly one category: `trivial-lint`, `trivial-format`, `trivial-import`, `trivial-generated-files`, `build-error`, `test-failure`, `infra-flake`, or `unknown`. +5. For trivial failures, identifies the specific files and (where possible) line numbers causing the issue. +6. Distinguishes infrastructure flakes (timeouts, network errors, pod scheduling failures, registry pull errors) from genuine code issues. +7. Produces a structured JSON analysis output per failed check containing: category, confidence level (high/medium/low), affected files, root cause summary, and suggested fix action. +8. Is a standalone executable script that accepts `--owner`, `--repo`, and `--pr-number` and reads CI status from `$RUNNER_TEMP/ci-status---.json`. + +### Dependencies + +Subtask 2 (needs the list of failed checks and their URLs from the CI status JSON). + +### Implementation Hints + +- **Log fetching (deterministic bash, before classification):** + ```bash + fetch_failure_logs() { + local pr_number="$1" + local status_file="${RUNNER_TEMP}/ci-status-${owner}-${repo}-${pr_number}.json" + + # gh pr checks output uses .bucket and .link fields + jq -r '.[] | select(.bucket == "fail") | .link' "$status_file" | while read -r url; do + if [[ "$url" == *"github.com"*"/actions/"* ]]; then + # GitHub Actions: extract run ID, fetch failed logs + run_id=$(echo "$url" | grep -oP 'runs/\K[0-9]+') + gh_retry gh run view "$run_id" --log-failed > "${RUNNER_TEMP}/log-${run_id}.txt" 2>/dev/null || true + elif [[ "$url" == *"prow.ci.openshift.org"* ]]; then + # Prow: target_url points to Prow UI (e.g., https://prow.ci.openshift.org/view/gs/BUCKET/PATH) + # Extract the GCS path and fetch build-log.txt from gcsweb + local gcs_path + gcs_path=$(echo "$url" | sed -n 's|.*/view/g[cs]s\?/||p') + if [[ -n "$gcs_path" ]]; then + local gcsweb_base="${GCSWEB_BASE_URL:-https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com}" + local gcsweb_url="${gcsweb_base}/gcs/${gcs_path}/build-log.txt" + # Prow build-log.txt can be 100K+ lines; truncate to last 1000 lines (failure is at the tail) + curl -sSL "$gcsweb_url" | tail -1000 > "${RUNNER_TEMP}/log-prow-$(date +%s).txt" 2>/dev/null || true + fi + fi + done + } + ``` +- **Deterministic classification (handles ~80-90% of cases, zero API cost):** + ```bash + classify_failure_deterministic() { + local log_file="$1" + local content + content=$(cat "$log_file") + + if echo "$content" | grep -qiE 'golangci-lint|golint|staticcheck|revive'; then + echo "trivial-lint" + elif echo "$content" | grep -qiE 'gofmt|goimports|formatting differs|diff.*\.go'; then + echo "trivial-format" + elif echo "$content" | grep -qiE 'imported and not used|could not import|import ordering'; then + echo "trivial-import" + elif echo "$content" | grep -qiE 'generated code is out of date|make generate|make manifests|deepcopy-gen|zz_generated'; then + echo "trivial-generated-files" + elif echo "$content" | grep -qiE 'cannot compile|undefined:|syntax error|cannot use.*as.*in'; then + echo "build-error" + elif echo "$content" | grep -qiE '--- FAIL|FAIL\s|panic:.*test|assertion failed'; then + echo "test-failure" + elif echo "$content" | grep -qiE 'context deadline exceeded|connection refused|i/o timeout|ErrImagePull|pod sandbox|TLS handshake timeout'; then + echo "infra-flake" + else + echo "unknown" + fi + } + ``` +- **Two-tier classification flow:** + ```bash + classify_failures() { + local pr_number="$1" + local log_dir="${RUNNER_TEMP}" + local results="[]" + local unknown_logs="" + + for log_file in "${log_dir}"/log-*.txt; do + [[ -f "$log_file" ]] || continue + local category + category=$(classify_failure_deterministic "$log_file") + + if [[ "$category" != "unknown" ]]; then + # Deterministic classification — no Claude API cost + results=$(echo "$results" | jq --arg cat "$category" --arg file "$log_file" \ + '. + [{"category": $cat, "confidence": "high", "affected_files": [], "root_cause": $cat, "suggested_fix": ""}]') + else + unknown_logs="${unknown_logs} ${log_file}" + fi + done + + # Fallback: invoke Claude only for unknown failures (read-only analysis) + if [[ -n "$unknown_logs" ]]; then + local claude_result + local classification_schema='{"type":"array","items":{"type":"object","properties":{"category":{"type":"string","enum":["trivial-lint","trivial-format","trivial-import","trivial-generated-files","build-error","test-failure","infra-flake","unknown"]},"confidence":{"type":"string","enum":["high","medium","low"]},"affected_files":{"type":"array","items":{"type":"string"}},"root_cause":{"type":"string"},"suggested_fix":{"type":"string"}},"required":["category","confidence","root_cause"]}}' + if ! claude_result=$(claude --print \ + --max-budget-usd "${MAX_BUDGET_PER_PR:-5.00}" \ + -p "$(cat plugins/oape/skills/ci-failure-analysis/SKILL.md) + Analyze the following CI failure logs and classify each failure. + $(for f in $unknown_logs; do echo "--- $(basename "$f") ---"; tail -1000 "$f"; done)" \ + --allowedTools "Bash(curl*),Read" \ + --json-schema "$classification_schema" 2>"${RUNNER_TEMP}/claude-stderr.txt"); then + audit_log "error" "claude-classification" "" "" \ + "Claude CLI failed: $(head -1 "${RUNNER_TEMP}/claude-stderr.txt")" + claude_result='[]' + fi + results=$(echo "$results" | jq --argjson cr "$claude_result" '. + $cr') + fi + + echo "$results" > "${RUNNER_TEMP}/failure-analysis-${owner}-${repo}-${pr_number}.json" + } + ``` +- **Classification heuristics (documented in the skill for Claude's guidance):** + - `trivial-lint`: Log contains `golangci-lint`, `golint`, `staticcheck`, or linter rule names. + - `trivial-format`: Log contains `gofmt`, `goimports`, `diff` output showing whitespace/formatting-only changes. + - `trivial-import`: Log contains `imported and not used`, `could not import`, or import ordering errors. + - `trivial-generated-files`: Log contains `generated code is out of date`, `make generate`, `make manifests`, `deepcopy-gen`. + - `build-error`: Log contains `cannot compile`, `undefined:`, `syntax error`, compilation errors. + - `test-failure`: Log contains `FAIL`, `--- FAIL`, test function names, assertion failures. + - `infra-flake`: Log contains `context deadline exceeded`, `connection refused`, `i/o timeout`, `ErrImagePull`, `pod sandbox`. +- **Skill structure:** Follow `plugins/oape/skills/analyze-rfe/SKILL.md` pattern — persona, prerequisites, step-by-step procedure. + +### Files + + +| File | Action | +| -------------------------------------------------- | ------------------------------------------------------------------------------------------- | +| `scripts/pr-agent/log-analyzer.sh` | Create (standalone executable: log fetching, deterministic classification, Claude fallback) | +| `plugins/oape/skills/ci-failure-analysis/SKILL.md` | Create | + + +--- + +## Subtask 4: Implement trivial auto-fix engine + +### Description + +Build the automated fix-and-push capability for trivial CI failures. When the failure analysis (Subtask 3) identifies a trivial issue, the agent checks out the PR branch, applies the appropriate fix command, verifies the fix compiles, and pushes. Auto-fix is always enabled in the CI job context (unlike the interactive mode which required `--auto-fix`). The `DRY_RUN` environment variable controls whether modifications are actually committed and pushed. + +### Acceptance Criteria + +1. In `DRY_RUN=true` mode, reports what *would* be fixed without modifying files or pushing. +2. Maps each trivial failure category to the correct fix command: + - `trivial-format` → `go fmt ./...` + - `trivial-import` → `goimports -w ` (or `go fmt ./...` if goimports unavailable) + - `trivial-lint` → targeted fix based on linter rule (e.g., `golangci-lint run --fix` where supported) + - `trivial-generated-files` → `make generate && make manifests` +3. Verifies fix compiles successfully (`go build ./...` and `go vet ./...`) before committing. +4. Creates a commit with a descriptive message following repository conventions (e.g., `fix: run goimports to resolve CI lint failure`). +5. Pushes to the PR's head branch using the GitHub App token (not `GITHUB_TOKEN`) to ensure CI is re-triggered. +6. Reports the fix to the audit log (commit SHA, files changed, fix type). +7. Respects all safety guardrails from Subtask 7 (file blocklist, commit limits, diff size limits). +8. **(Phase 2)** For `infra-flake` failures: posts a targeted `/test ` comment to re-trigger only the flaky job (not blanket `/retest`). Gated by `RETEST_INFRA_FLAKES` config flag (default `false`). Limited to max 2 retests per job per run to prevent retry loops. Tracked in the state to avoid re-posting on subsequent runs. + +### Dependencies + +Subtask 3 (needs failure classification to determine fix type and affected files). +Subtask 7 (safety guardrails must be enforced before any file modification). + +### Implementation Hints + +- **Checkout and fix flow:** + ```bash + apply_trivial_fixes() { + local owner="$1" repo="$2" pr_number="$3" + local analysis_file="${RUNNER_TEMP}/failure-analysis-${owner}-${repo}-${pr_number}.json" + local pr_commit_count=0 + # Global commit counter file shared across auto-fix and review handler + local commit_counter_file="${RUNNER_TEMP}/pr-agent-commit-count.txt" + local total_commits + total_commits=$(cat "$commit_counter_file" 2>/dev/null || echo 0) + + # Clone with blobless filter for performance (OpenShift repos can be multi-GB) + local workdir="${RUNNER_TEMP}/repo-${owner}-${repo}-${pr_number}" + gh repo clone "${owner}/${repo}" "$workdir" -- --filter=blob:none --single-branch + cd "$workdir" + gh pr checkout "$pr_number" + + # Configure git identity for the bot (matches the GitHub App identity) + git config user.name "openshift-app-platform-shift-bot" + git config user.email "267347085+openshift-app-platform-shift-bot@users.noreply.github.com" + git remote set-url origin "https://x-access-token:${GH_TOKEN}@github.com/${owner}/${repo}.git" + + while read -r fix; do + local category=$(echo "$fix" | jq -r '.category') + local files=$(echo "$fix" | jq -r '.affected_files[]') + + # Pre-fix blocklist check (fast guard on known affected files, category-aware for go.sum exception) + if ! check_blocklist "$files" "$category"; then + audit_log "blocked" "$category" "$files" "" "security-sensitive file" + continue + fi + + # Check global commit limits + if [[ "$total_commits" -ge "${MAX_COMMITS_PER_RUN:-10}" ]]; then + audit_log "skipped" "$category" "$files" "" "run commit limit reached" + continue + fi + if [[ "$pr_commit_count" -ge "${MAX_COMMITS_PER_PR:-3}" ]]; then + audit_log "skipped" "$category" "$files" "" "per-PR commit limit reached" + continue + fi + + # Apply the fix (framework-aware for generated files) + # Determine PR base branch for scoping fixes to PR-changed files only + local base_branch + base_branch=$(gh pr view "$pr_number" --repo "${owner}/${repo}" --json baseRefName -q .baseRefName) + git fetch origin "${base_branch}" --depth=1 2>/dev/null || true + + case "$category" in + trivial-format) + git diff --name-only HEAD "$(git merge-base HEAD "origin/${base_branch}")" -- '*.go' | xargs -r go fmt + ;; + trivial-import) goimports -w $files ;; + trivial-lint) golangci-lint run --fix ./... 2>/dev/null || true ;; + trivial-generated-files) + if grep -q 'sigs.k8s.io/controller-runtime' go.mod; then + make generate && make manifests + elif grep -q 'github.com/openshift/library-go' go.mod; then + make update + else + make generate 2>/dev/null || make update 2>/dev/null || true + fi + ;; + esac + + # Verify fix compiles + if ! go build ./... || ! go vet ./...; then + git checkout -- . + git clean -fd + audit_log "reverted" "$category" "$files" "" "fix broke compilation" + continue + fi + + # Post-fix blocklist check (safety net — verify ACTUAL modified files) + local modified_files + modified_files=$(git diff --name-only; git ls-files --others --exclude-standard) + if ! check_blocklist "$modified_files" "$category"; then + git checkout -- . + git clean -fd + audit_log "reverted" "$category" "$modified_files" "" "post-fix blocklist violation" + continue + fi + + # Check diff size guard (count both insertions and deletions) + local diff_lines + diff_lines=$(git diff --numstat | awk '{s+=$1+$2} END {print s+0}') + if [[ "$diff_lines" -gt 500 ]]; then + git checkout -- . + git clean -fd + audit_log "reverted" "$category" "$files" "" "diff too large ($diff_lines lines)" + continue + fi + + if [[ "${DRY_RUN:-false}" == "true" ]]; then + audit_log "dry-run" "$category" "$files" "" "would commit and push" + git checkout -- . + git clean -fd + continue + fi + + # Stage both modified tracked files AND new untracked files + git diff --name-only -z | xargs -0 git add + git ls-files --others --exclude-standard -z | xargs -0 git add + git commit -m "fix: ${category} — auto-fix by oape-pr-agent" + local sha=$(git rev-parse HEAD) + + # Pull before push to handle concurrent pushes to the same branch + if ! git pull --rebase origin HEAD 2>/dev/null; then + git rebase --abort 2>/dev/null || true + audit_log "reverted" "$category" "$files" "$sha" "rebase conflict — concurrent push detected" + git reset --hard HEAD~1 + continue + fi + git push origin HEAD + pr_commit_count=$((pr_commit_count + 1)) + total_commits=$((total_commits + 1)) + echo "$total_commits" > "$commit_counter_file" + + audit_log "auto-fix" "$category" "$files" "$sha" "success" + done < <(jq -c '.[] | select(.category | startswith("trivial-"))' "$analysis_file") + } + ``` +- **GitHub App token for push:** The token generated via JWT signing from the Prow-mounted PEM key is set as `GH_TOKEN` and also used for git push via: + ```bash + git remote set-url origin "https://x-access-token:${GH_TOKEN}@github.com/${owner}/${repo}.git" + ``` +- **Reference:** `plugins/oape/commands/implement-review-fixes.md` for the fix-verify-commit pattern already used in OAPE. HyperShift's Jira Agent uses a similar clone → fix → push → PR flow. + +### Files + + +| File | Action | +| ------------------------------ | --------------------------------------------------------- | +| `scripts/pr-agent/auto-fix.sh` | Create (standalone executable, called by `entrypoint.sh`) | + + +--- + +## Subtask 5: Implement review comment monitoring and response + +### Description + +Add the ability to fetch, analyze, and respond to review comments on PRs in allowed repos. Following HyperShift's Review Agent pattern, the agent identifies unresolved review threads that need attention, skips threads already addressed by the bot, and invokes Claude Code CLI to generate appropriate responses (code changes for actionable requests, explanations for questions). Bot-generated comments are filtered via `SKIP_USERS`. + +### Acceptance Criteria + +1. Fetches all review threads (inline and top-level) and review summaries from the PR. +2. Implements HyperShift-style comment analysis logic: + - **Process**: No bot reply in thread (first response needed), or human replied after bot's last comment (follow-up needed). + - **Skip**: Bot already replied with no human follow-up, thread is resolved, thread is outdated (code changed). +3. Filters out bot-generated comments using a configurable skip list (default: `openshift-ci`, `openshift-bot`, `dependabot`, `codecov`, `sonarcloud`, `coderabbitai[bot]`). Additional users configured via `SKIP_USERS` env var. +4. Skips known bot accounts (via `SKIP_USERS`). All other commenters are treated as legitimate reviewers — branch protection and repo permissions provide the authorization boundary. +5. Invokes Claude Code CLI to address each unresolved thread. Claude receives the full thread context and decides whether to make code changes or provide an explanation — no separate intent classification step. The safety skill content is included via `cat` in the prompt. +6. Pushes code changes (if any) and posts inline reply comments via `gh api`. +7. Respects the global commit counter shared with the auto-fix engine (via `$RUNNER_TEMP/pr-agent-commit-count.txt`). Increments the counter for each commit pushed. +8. Claude Code invocations use `--allowedTools` to exclude destructive git operations (no `git push --force`, `git push -f`, `git rebase`, `git reset --hard`). + +### Dependencies + +Subtask 1 (entrypoint must provide PR context). + +### Implementation Hints + +- **Thread analysis (deterministic bash):** + ```bash + analyze_review_threads() { + local owner="$1" repo="$2" pr_number="$3" + + # Fetch review comments (inline) + local comments + comments=$(gh api "repos/${owner}/${repo}/pulls/${pr_number}/comments" \ + --paginate --jq '.') + + # Fetch review summaries + local reviews + reviews=$(gh api "repos/${owner}/${repo}/pulls/${pr_number}/reviews" \ + --paginate --jq '.') + + # Fetch top-level PR conversation comments (not inline on code) + local issue_comments + issue_comments=$(gh api "repos/${owner}/${repo}/issues/${pr_number}/comments" \ + --paginate --jq '.') + + # Group by thread (in_reply_to_id), determine if bot has replied + # Filter: skip resolved, skip outdated, skip unauthorized authors + # Output: list of threads needing attention + } + ``` +- **Bot detection:** + ```bash + SKIP_USERS="${SKIP_USERS:-openshift-ci,openshift-bot,dependabot,codecov,sonarcloud,coderabbitai[bot]}" + is_bot_or_skipped() { + local login="$1" user_type="$2" + [[ "$user_type" == "Bot" ]] && return 0 + echo "$SKIP_USERS" | tr ',' '\n' | grep -qx "$login" && return 0 + return 1 + } + ``` +- **Claude Code CLI invocation for review response:** +Claude receives the full thread context and decides whether to make code changes or provide an explanation. No separate `classify_thread_intent()` step — Claude handles this naturally based on the comment content. + ```bash + address_review_thread() { + local owner="$1" repo="$2" pr_number="$3" thread_file="$4" + local workdir="${RUNNER_TEMP}/repo-${owner}-${repo}-${pr_number}" + + cd "$workdir" + + # Check global commit limit before invoking Claude + local commit_counter_file="${RUNNER_TEMP}/pr-agent-commit-count.txt" + local total_commits + total_commits=$(cat "$commit_counter_file" 2>/dev/null || echo 0) + if [[ "$total_commits" -ge "${MAX_COMMITS_PER_RUN:-10}" ]]; then + echo "[review] Skipping thread — commit limit reached" + return 0 + fi + + # Include safety skill content and let Claude decide how to respond + claude --print \ + --max-budget-usd "${MAX_BUDGET_PER_PR:-5.00}" \ + -p "$(cat plugins/oape/skills/pr-agent-safety/SKILL.md) + + Address the following review comment on PR #${pr_number} in ${owner}/${repo}. + Review thread: $(cat "$thread_file") + + If the reviewer requests a code change (imperative language like 'change', 'fix', 'update', + 'remove', 'add'), make the change, verify it compiles (go build ./...), and commit. + If the reviewer asks a question, reply with a concise explanation only — do NOT change code. + One response per feedback — never respond via both inline reply AND general PR comment." \ + --allowedTools "Bash(git diff*),Bash(git add*),Bash(git commit*),Bash(git push origin HEAD),Bash(git log*),Bash(git status*),Bash(go*),Bash(make*),Bash(gh api*),Bash(gh pr comment*),Read,Write,Edit" + + # Update global commit counter if Claude pushed commits + local new_commits + new_commits=$(git rev-list --count HEAD ^"${HEAD_SHA_BEFORE}") + if [[ "$new_commits" -gt 0 ]]; then + total_commits=$((total_commits + new_commits)) + echo "$total_commits" > "$commit_counter_file" + fi + } + ``` + > **Note:** The `--allowedTools` restriction explicitly excludes `git push --force`, + > `git push -f`, `git rebase`, and `git reset --hard` — only safe git operations are + > permitted. The existing `/oape:implement-review-fixes` command pattern is referenced + > in the prompt for fix prioritization and verification patterns. +- **Reference:** HyperShift's Review Agent (`periodic-review-agent`) uses identical thread analysis logic. Their `/utils:address-reviews` command is the equivalent of this Claude Code invocation. + +### Files + + +| File | Action | +| ------------------------------------ | --------------------------------------------------------- | +| `scripts/pr-agent/review-handler.sh` | Create (standalone executable, called by `entrypoint.sh`) | + + +--- + +## Subtask 6: Wire together the PR processing pipeline + +### Description + +Create the `process_pr()` orchestration function in `entrypoint.sh` that wires together all capabilities from Subtasks 2–5 and 7–8 into the end-to-end processing pipeline. This subtask does NOT modify the workflow YAML files (those are fully specified in Subtask 0) — it only adds the function that sequences CI monitoring → failure analysis → auto-fix → review handling → status reporting for a single PR. + +> **Phase 1 implementation:** `process_pr()` checks the `MONITOR_ONLY` environment variable (set by the `--monitor-only` CLI flag). When `MONITOR_ONLY=true`, the auto-fix phase is skipped entirely — the pipeline runs: merge conflict check → CI monitoring → failure classification → status report. This allows Phase 1 to validate CI monitoring and reporting without risk of pushing code changes. + +### Acceptance Criteria + +1. The `process_pr()` function invokes all phases in order: CI status check → failure log analysis → trivial auto-fix → review comment handling → status report. +2. Phases are conditional: failure analysis and auto-fix only run when CI status is `some-failed`; review comment handling only runs when there are unresolved threads. +3. Each phase logs its start/end with structured output: `[PR #N] Phase: — started/completed`. +4. If a phase fails, it logs the error and continues to the next phase (best-effort processing). +5. The `run_periodic()` loop and `run_on_demand()` entry point call `process_pr()` for each PR. +6. Each PR is processed with a per-PR time limit (`PR_TIMEOUT_SECONDS`, default 720 = 12 minutes). On timeout, posts a partial status report and continues to the next PR, preventing one complex PR from starving subsequent PRs. + +### Dependencies + +Subtasks 1, 2, 3, 4, 5, 7, 8 (all capabilities must exist before they can be wired together). + +### Implementation Hints + +- **Periodic mode main loop** (in `entrypoint.sh`): + ```bash + run_periodic() { + local max_prs="${PR_AGENT_MAX_PRS:-4}" + local processed=0 + + discover_oape_prs # Populates $PR_LIST_FILE + + while IFS= read -r pr_url; do + if [[ "$processed" -ge "$max_prs" ]]; then + echo "[periodic] Reached max PRs ($max_prs), stopping" + break + fi + + echo "[periodic] Processing PR $((processed + 1))/${max_prs}: $pr_url" + local pr_timeout="${PR_TIMEOUT_SECONDS:-720}" + if timeout "$pr_timeout" bash -c "process_pr '$pr_url'"; then + echo "[periodic] PR $pr_url — completed successfully" + elif [[ $? -eq 124 ]]; then + echo "[periodic] PR $pr_url — timed out after ${pr_timeout}s, posting partial report" + parse_pr_url "$pr_url" + scripts/pr-agent/report.sh --owner "$OWNER" --repo "$REPO" --pr-number "$PR_NUMBER" --partial + else + echo "[periodic] PR $pr_url — failed (continuing to next)" + fi + + processed=$((processed + 1)) + + # Rate limit between PRs + if [[ "$processed" -lt "$max_prs" ]]; then + echo "[periodic] Waiting 60s before next PR..." + sleep 60 + fi + done < "$PR_LIST_FILE" + + echo "[periodic] Processed $processed PRs" + } + ``` +- **On-demand mode** (in `entrypoint.sh`): + ```bash + run_on_demand() { + local pr_url="$1" + echo "[on-demand] Processing single PR: $pr_url" + process_pr "$pr_url" + echo "[on-demand] Done" + } + ``` +- **Process function** (invokes all phases via standalone scripts): + ```bash + process_pr() { + local pr_url="$1" + parse_pr_url "$pr_url" + + # Phase 0: Merge conflict check + local mergeable + mergeable=$(gh_retry gh pr view "$pr_url" --json mergeable -q .mergeable) + if [[ "$mergeable" == "CONFLICTING" ]]; then + echo "[PR #${PR_NUMBER}] Merge conflict detected — skipping to report" + scripts/pr-agent/report.sh --owner "$OWNER" --repo "$REPO" \ + --pr-number "$PR_NUMBER" --merge-conflict + return 0 + fi + + # Phase 1: CI Check Monitoring (standalone script — outputs aggregate status to stdout, saves details to JSON) + local ci_status + ci_status=$(scripts/pr-agent/ci-monitor.sh --owner "$OWNER" --repo "$REPO" --pr-number "$PR_NUMBER") + echo "[CI] Status: $ci_status" + + # Phase 2: Failure Analysis + Auto-Fix (only if failures exist) + if [[ "$ci_status" == "some-failed" ]]; then + scripts/pr-agent/log-analyzer.sh --owner "$OWNER" --repo "$REPO" --pr-number "$PR_NUMBER" + scripts/pr-agent/auto-fix.sh --owner "$OWNER" --repo "$REPO" --pr-number "$PR_NUMBER" + fi + + # Phase 3: Review Comment Handling (standalone script) + scripts/pr-agent/review-handler.sh --owner "$OWNER" --repo "$REPO" --pr-number "$PR_NUMBER" + + # Phase 4: Status Report (standalone script) + scripts/pr-agent/report.sh --owner "$OWNER" --repo "$REPO" --pr-number "$PR_NUMBER" + } + ``` +- **Reference:** HyperShift's `periodic-review-agent` runs every 3 hours and processes up to 10 PRs. The `address-review-comments` job is the on-demand equivalent. Both share setup steps and use the same processing logic. + +### Files + + +| File | Action | +| -------------------------------- | -------------------------------------------------------------------------- | +| `scripts/pr-agent/entrypoint.sh` | Modify (add `process_pr()`, `run_periodic()`, `run_on_demand()` functions) | + + +--- + +## Subtask 7: Implement safety guardrails and file-modification boundaries + +### Description + +Define and enforce safety boundaries for the autonomous agent. Since the agent can modify code and push to branches in a CI context, strong guardrails are essential to prevent accidental damage. This includes file blocklists, commit limits, diff size limits, force-push prevention, and a comprehensive audit log. A dedicated safety script encapsulates the guardrail functions, and a Claude Code skill documents the safety rules for the LLM's awareness. + +### Acceptance Criteria + +1. Maintains a blocklist of file patterns that are never auto-modified. Uses extension-aware patterns to protect actual secret storage files while allowing Go source files that operate on Kubernetes Secret/Token resources: + - Secret storage files: `*.key`, `*.pem`, `*.crt`, `*.cert`, `*.p12`, `*.pfx`, `*.env`, `credentials.`*, `kubeconfig` + - Container/CI files: `Dockerfile`, `Containerfile`, `.dockerignore` + - Workflow/build files: `.github/workflows/*`, `.tekton/*`, `Makefile` + - RBAC manifests: `**/rbac/*.yaml`, `**/clusterrole*.yaml` + - Dependency files: `go.mod` (blocked by default, but **allowed for the `trivial-generated-files` category** since `make generate` legitimately modifies it via `go mod tidy`) + - `go.sum` (blocked by default, but **allowed for the `trivial-generated-files` category** since `make generate` legitimately modifies it via `go mod tidy`) +2. Enforces commit limits: max 3 commits per PR processing, max 10 total commits across all PRs in a single run. Stops auto-fixing (but continues monitoring/reporting) when limits are reached. +3. Never executes `git push --force` or any destructive operation that modifies remote/shared history. Local rollback of unpushed agent commits (e.g., `git reset --hard HEAD~1` after a failed rebase, `git pull --rebase` to sync with concurrent pushes) is permitted as a recovery mechanism. +4. Logs every action to a structured audit log (JSON lines format) at `$RUNNER_TEMP/pr-agent-audit-.jsonl` including: timestamp, PR URL, action type, affected files, commit SHA (if applicable), and outcome. +5. `DRY_RUN=true` mode executes the full analysis pipeline but skips all file modifications, commits, and pushes. Reports what *would* have been done. +6. Diff size guard: if an auto-fix produces more than 500 lines of changes, abort and report. +7. Audit log is available in Prow GCS artifacts at the end of every run. + +### Dependencies + +None — this is a standalone utility module. Its functions are consumed by Subtasks 4, 5, and 6 but it has no dependency on them. + +### Implementation Hints + +- **Blocklist check function (category-aware for go.sum exception):** + ```bash + # Protect actual secret storage files (not Go source files that operate on Secrets) + BLOCKED_PATTERNS='\.(key|pem|crt|cert|p12|pfx)$|\.env$|credentials\.|(^|/)kubeconfig$' + BLOCKED_PATTERNS+='|(^|/)Dockerfile$|(^|/)Containerfile$|\.dockerignore$' + BLOCKED_PATTERNS+='|\.github/workflows|\.tekton/|(^|/)Makefile$' + BLOCKED_PATTERNS+='|rbac/.*\.yaml|clusterrole.*\.yaml' + BLOCKED_PATTERNS+='|go\.mod|go\.sum' + # Same patterns but without go.mod and go.sum (allowed for trivial-generated-files) + BLOCKED_PATTERNS_GENERATED='\.(key|pem|crt|cert|p12|pfx)$|\.env$|credentials\.|(^|/)kubeconfig$' + BLOCKED_PATTERNS_GENERATED+='|(^|/)Dockerfile$|(^|/)Containerfile$|\.dockerignore$' + BLOCKED_PATTERNS_GENERATED+='|\.github/workflows|\.tekton/|(^|/)Makefile$' + BLOCKED_PATTERNS_GENERATED+='|rbac/.*\.yaml|clusterrole.*\.yaml' + + check_blocklist() { + local files="$1" + local category="${2:-}" + local patterns="$BLOCKED_PATTERNS" + # Allow go.mod and go.sum for trivial-generated-files (make generate legitimately modifies them via go mod tidy) + if [[ "$category" == "trivial-generated-files" ]]; then + patterns="$BLOCKED_PATTERNS_GENERATED" + fi + if echo "$files" | grep -iqE "$patterns"; then + return 1 # blocked + fi + return 0 # safe + } + ``` +- **Audit log function:** + ```bash + AUDIT_LOG="${RUNNER_TEMP}/pr-agent-audit-${GITHUB_RUN_ID:-local}.jsonl" + + audit_log() { + local action="$1" category="$2" files="$3" commit="$4" outcome="$5" + local ts + ts=$(date -u +"%Y-%m-%dT%H:%M:%SZ") + printf '{"ts":"%s","pr":"%s","action":"%s","type":"%s","files":%s,"commit":"%s","outcome":"%s"}\n' \ + "$ts" "${CURRENT_PR_URL:-}" "$action" "$category" \ + "$(echo "$files" | jq -R 'split(" ")' 2>/dev/null || echo '[]')" \ + "$commit" "$outcome" \ + >> "$AUDIT_LOG" + } + ``` +- **Commit counter (global across the run):** + ```bash + TOTAL_COMMITS=0 + MAX_COMMITS_PER_RUN="${MAX_COMMITS_PER_RUN:-10}" + MAX_COMMITS_PER_PR=3 + + check_commit_limit() { + if [[ "$TOTAL_COMMITS" -ge "$MAX_COMMITS_PER_RUN" ]]; then + echo "GUARDRAIL: Total commit limit reached ($TOTAL_COMMITS/$MAX_COMMITS_PER_RUN)" + return 1 + fi + return 0 + } + ``` +- **Diff size guard:** + ```bash + check_diff_size() { + local max_lines="${MAX_DIFF_LINES:-500}" + local changed_lines + changed_lines=$(git diff --numstat | awk '{s+=$1+$2} END {print s+0}') + if [[ "$changed_lines" -gt "$max_lines" ]]; then + echo "GUARDRAIL: Diff too large ($changed_lines lines > $max_lines limit)" + return 1 + fi + return 0 + } + ``` +- **Skill for Claude's awareness:** + ```markdown + # Safety Guardrails for PR Agent + + When operating as the OAPE PR agent, you MUST follow these rules: + 1. NEVER modify files matching: [blocklist patterns] + 2. NEVER use git push --force, git rebase, or git reset --hard + 3. ALWAYS verify changes compile before committing + 4. STOP if diff exceeds 500 lines + ``` +- **Reference:** KNOWN-ISSUES.md documents "Unrestricted agent permissions" as a critical issue. HyperShift's agents enforce similar guardrails: "Cannot execute destructive operations — no ability to delete resources or force-push." + +### Files + + +| File | Action | +| ---------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- | +| `scripts/pr-agent/safety.sh` | Create (sourced utility library, provides shared functions to entrypoint.sh, auto-fix.sh, review-handler.sh, and report.sh) | +| `plugins/oape/skills/pr-agent-safety/SKILL.md` | Create | + + +--- + +## Subtask 8: Implement status reporting and PR comment summary + +### Description + +Build the reporting layer that gives developers clear visibility into what the agent did. The agent produces a structured markdown report and posts it as a PR comment, so developers see results directly in the PR conversation. The report is also available in Prow GCS artifacts for archival. Following HyperShift's pattern, the report includes token/cost tracking data. + +### Acceptance Criteria + +1. Produces a markdown summary report containing all of the following sections: + - **PR Status:** current state, branch, title, URL. + - **Merge Conflict Status:** if the PR has merge conflicts, prominently flagged as the primary action item. When merge conflicts are detected, CI analysis and auto-fix sections are replaced with a message directing the developer to resolve conflicts first. + - **CI Check Results:** table of all checks with pass/fail/pending status. + - **Fixes Applied:** list of auto-fixes with commit SHA, fix type, and files changed. + - **Review Comments Addressed:** summary of review threads handled. + - **Infrastructure Flakes:** list of CI jobs classified as infrastructure flakes (timeouts, network errors, registry pull failures) with job names and links. Presented separately from code failures so developers can quickly identify retestable jobs. + - **Remaining Issues:** items requiring manual intervention, separated into "auto-fixable but blocked by guardrails" vs. "requires human judgment." + - **Run Summary:** total time elapsed, commit count. (Claude API costs are tracked at the GCP project billing level via Vertex AI, not per-invocation.) +2. Each auto-fix entry includes a clickable link to the commit on GitHub (`https://github.com/{owner}/{repo}/commit/{sha}`). +3. Report is posted as a PR comment via `gh pr comment`. If a previous agent comment exists, it is updated (not duplicated). +4. Report is saved to `$RUNNER_TEMP/pr-agent-report---.md` and available in Prow GCS artifacts. +5. When `DRY_RUN=true`, the report clearly states it was a dry run and no changes were made. +6. Before pushing any auto-fixes, posts an "in progress" comment (or updates the existing report comment with a "Processing..." header) so developers see context before surprise commits appear on the branch. The final report replaces this in-progress state. +7. On agent crash or failure, a `trap` handler posts a brief error note to the PR comment so developers know the agent attempted but failed. + +### Dependencies + +Subtasks 2–7 (aggregates data from all other capabilities). + +### Implementation Hints + +- **Report generation:** + ```bash + generate_status_report() { + local owner="$1" repo="$2" pr_number="$3" + local report_file="${RUNNER_TEMP}/pr-agent-report-${owner}-${repo}-${pr_number}.md" + local audit_file="${RUNNER_TEMP}/pr-agent-audit-${GITHUB_RUN_ID:-local}.jsonl" + local ci_file="${RUNNER_TEMP}/ci-status-${owner}-${repo}-${pr_number}.json" + + local pr_info + pr_info=$(gh pr view "$pr_number" --repo "${owner}/${repo}" \ + --json title,url,headRefName,baseRefName -q '.') + + local title=$(echo "$pr_info" | jq -r '.title') + local url=$(echo "$pr_info" | jq -r '.url') + local head=$(echo "$pr_info" | jq -r '.headRefName') + local base=$(echo "$pr_info" | jq -r '.baseRefName') + + local passed failed pending + passed=$(jq '[.[] | select(.bucket == "pass")] | length' "$ci_file") + failed=$(jq '[.[] | select(.bucket == "fail")] | length' "$ci_file") + pending=$(jq '[.[] | select(.bucket == "pending")] | length' "$ci_file") + + local fixes_applied + fixes_applied=$(grep '"action":"auto-fix"' "$audit_file" 2>/dev/null | wc -l || echo 0) + + cat > "$report_file" </dev/null | jq -r '"- [\(.commit)](https://github.com/'${owner}'/'${repo}'/commit/\(.commit)) — `\(.type)`: \(.outcome)"' || echo "- (none)") + + ### Remaining Issues + $(grep '"action":"blocked\|"action":"skipped"' "$audit_file" 2>/dev/null | jq -r '"- `\(.type)` on \(.files | join(", ")): \(.outcome)"' || echo "- (none)") + + --- + *Generated by oape-pr-agent on $(date -u +"%Y-%m-%d %H:%M UTC")* + EOF + } + ``` +- **Post as PR comment (update if exists):** + ```bash + post_status_comment() { + local owner="$1" repo="$2" pr_number="$3" + local report_file="${RUNNER_TEMP}/pr-agent-report-${owner}-${repo}-${pr_number}.md" + local marker="" + + # Check for existing agent comment and load persisted state + local existing_comment_id existing_body + existing_comment_id=$(gh api "repos/${owner}/${repo}/issues/${pr_number}/comments" \ + --jq ".[] | select(.body | contains(\"${marker}\")) | .id" | head -1) + if [[ -n "$existing_comment_id" ]]; then + existing_body=$(gh api "repos/${owner}/${repo}/issues/comments/${existing_comment_id}" --jq .body) + # Extract and restore persisted state from previous run + local persisted_state + persisted_state=$(echo "$existing_body" | grep -oP '(?<=oape-pr-agent-state:)[A-Za-z0-9+/=]+' | head -1) + if [[ -n "$persisted_state" ]]; then + echo "$persisted_state" | base64 -d > "${RUNNER_TEMP}/pr-agent-state-${owner}-${repo}-${pr_number}.json" + fi + fi + + # Embed cross-run state in the comment for persistence across job runs + local state_file="${RUNNER_TEMP}/pr-agent-state-${owner}-${repo}-${pr_number}.json" + local state_block="" + if [[ -f "$state_file" ]]; then + local state_b64 + state_b64=$(base64 -w0 < "$state_file") + state_block="" + fi + + local body="${marker}${state_block} + $(cat "$report_file")" + + if [[ -n "$existing_comment_id" ]]; then + gh api "repos/${owner}/${repo}/issues/comments/${existing_comment_id}" \ + -X PATCH -f body="$body" + else + gh pr comment "$pr_number" --repo "${owner}/${repo}" --body "$body" + fi + } + ``` +- **Reference:** HyperShift's Dependabot Triage Agent generates an HTML report with token usage and cost breakdown. The OAPE report follows the same principle but in markdown. + +### Files + + +| File | Action | +| ---------------------------- | --------------------------------------------------------- | +| `scripts/pr-agent/report.sh` | Create (standalone executable, called by `entrypoint.sh`) | + + +--- + +## Subtask 9: PR agent testing and validation + +### Description + +Add automated testing for the PR agent itself. Since the agent autonomously pushes code to production repositories, it must be validated before deployment. This includes static analysis of bash scripts, a dry-run integration test against a known-state test PR, and a CI workflow that runs validation on every push to this repo. + +### Acceptance Criteria + +1. All bash scripts in `scripts/pr-agent/*.sh` pass **shellcheck** with zero errors and zero warnings. +2. A **dry-run integration test** script exists that: + - Creates a test PR in a designated test repository (or uses a pre-existing test PR). + - Runs the full agent pipeline in `DRY_RUN=true` mode. + - Verifies: PR discovery finds the test PR, CI status is fetched, failure analysis produces valid JSON, report is generated (but not posted). + - Exits with a non-zero status if any phase fails. +3. A **CI validation target** (Makefile or Prow presubmit) runs shellcheck on all `scripts/pr-agent/*.sh` and `scripts/ci-monitor/*.sh` files and executes the dry-run integration test. +4. Test scripts themselves follow shellcheck-clean conventions. + +### Dependencies + +Subtasks 0–8 (all agent components must exist before they can be tested). + +### Implementation Hints + +- **Shellcheck in CI:** + ```yaml + - name: Lint bash scripts + run: | + shellcheck scripts/pr-agent/*.sh + ``` +- **Dry-run integration test:** + ```bash + #!/usr/bin/env bash + set -euo pipefail + # Run the full agent pipeline against a known test PR in dry-run mode + export DRY_RUN=true + export PR_AGENT_MAX_PRS=1 + + # Use a pre-existing test PR (created once, kept open for testing) + TEST_PR_URL="${TEST_PR_URL:-https://github.com/openshift-eng/oape-ai-e2e/pull/1}" + + scripts/pr-agent/entrypoint.sh --mode on-demand --pr-url "$TEST_PR_URL" + + # Verify outputs were generated (owner-repo-pr_number naming convention) + [[ -f "${RUNNER_TEMP}/ci-status-openshift-eng-oape-ai-e2e-1.json" ]] || { echo "FAIL: CI status not generated"; exit 1; } + [[ -f "${RUNNER_TEMP}/pr-agent-report-openshift-eng-oape-ai-e2e-1.md" ]] || { echo "FAIL: Report not generated"; exit 1; } + + echo "PASS: Dry-run integration test completed successfully" + ``` +### Files + + +| File | Action | +| ------------------------------------- | ------ | +| `scripts/pr-agent/test-dry-run.sh` | Create | + + +--- + +## Subtask 10: Create `/oape:pr-agent` command + +### Description + +Create a Claude Code command that serves as the **interactive/developer** entry point for the PR agent, following the pattern of all existing OAPE commands (`/oape:review`, `/oape:init`, etc.). This command is for developers running the agent locally in their terminal — the Prow presubmit invokes `monitor.sh` and `dispatch.sh` directly (deterministic, no Claude orchestration overhead). This separation ensures the CI path is fast and predictable, while the interactive path provides a richer developer experience. + +### Acceptance Criteria + +1. A command file exists at `plugins/oape/commands/pr-agent.md` following the project's command pattern (frontmatter with `description` and `argument-hint`, Synopsis, Description, Arguments, Implementation sections). +2. Accepts a PR URL as the primary argument. +3. Supports flags: `--dry-run` (no modifications), `--auto-fix` (default true). +4. Prompts the user before pushing fixes, displays status inline, and offers to monitor the PR on a schedule (via `CronCreate`). +5. Delegates to the same bash scripts (`ci-monitor.sh`, `auto-fix.sh`, etc.) used by the Prow presubmit, ensuring parity between interactive and CI execution. +6. The CLAUDE.md command table is updated to include `/oape:pr-agent`. +7. **Note:** The Prow presubmit invokes `monitor.sh` and `dispatch.sh` directly — it does NOT invoke this command. This command is for developer use only. + +### Dependencies + +Subtasks 1–8 (the command wraps all existing capabilities). + +### Implementation Hints + +- **Command frontmatter pattern** (follow `plugins/oape/commands/review.md`): + ```markdown + --- + description: Monitor a PR, auto-fix trivial CI failures, address review comments, and report status + argument-hint: [--dry-run] [--auto-fix] + --- + ``` +- **Interactive monitoring** uses `CronCreate` for periodic re-checks (like yolo-agent's interactive mode): + ``` + After the initial pass, offer: "Would you like me to keep monitoring this PR?" + If yes, schedule a one-shot CronCreate to re-run the analysis in 5 minutes. + ``` +- **Prow path is separate:** The Prow presubmit invokes `monitor.sh` and `dispatch.sh` directly — it does not use this command. This keeps the CI path deterministic and avoids Claude orchestration overhead. + +### Files + + +| File | Action | +| ----------------------------------- | ------ | +| `plugins/oape/commands/pr-agent.md` | Create | + + +--- + +## Data Flow and Security + +### Authentication + + +| System | Method | Details | +| ------------------- | ---------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| GitHub (read/write) | GitHub App token with `GITHUB_TOKEN` fallback | **Primary:** App token generated via JWT signing from PEM key at `/var/run/github-app/private-key.pem` (Prow secret: `openshift-app-platform-shift-github-bot`). Required for Phase 2+ pushes (avoids `GITHUB_TOKEN` anti-recursion). **Fallback:** If App is not installed on the target repo, uses `GITHUB_TOKEN` (sufficient for Phase 1 read + comment). | +| Claude API | GCP Application Default Credentials (ADC) via Vertex AI | `CLAUDE_CODE_USE_VERTEX=1`, `ANTHROPIC_VERTEX_PROJECT_ID=itpc-gcp-hcm-pe-eng-claude`. ADC JSON mounted at `/var/run/gcloud-adc/application_default_credentials.json` (Prow secret: `oap-lts-claude-gcp-vertex-sa`). | +| GitHub (CI logs) | Same GitHub App installation token | Used via `gh` CLI for `gh api` calls and log fetching. | +| Sippy API | None (unauthenticated) | Public API at `sippy.dptools.openshift.org`. Used by `monitor.sh` to query flake history for test failures. No credentials required. | + + +### Data Retention + +- No persistent storage beyond PR comments and Prow GCS artifacts. +- Audit logs and reports are written to the Prow job container's filesystem and available in GCS build artifacts for the job's retention period (standard OpenShift CI retention). +- No secrets are logged — the audit log contains only file paths, commit SHAs, and action outcomes. + +### Data Flow: Prow Presubmit CI Monitor + +``` + Developer pushes to PR branch + → Prow triggers oape-ci-monitor presubmit (alongside other CI jobs) + → Also triggerable manually via: /test oape-ci-monitor + +┌─────────────────────────────────────────────────────────────────┐ +│ Prow Pod (ci-monitor-agent container, ci-operator managed) │ +│ │ +│ ┌───────────┐ ┌───────────────┐ ┌─────────────────────┐ │ +│ │ Generate │───▶│ monitor.sh │───▶│ dispatch.sh │ │ +│ │ GitHub │ │ (poll checks, │ │ (log trigger actions,│ │ +│ │ App token │ │ collect GCS │ │ Phase 2+: invoke │ │ +│ │ from PEM │ │ artifacts, │ │ auto-fix/Claude) │ │ +│ └───────────┘ │ classify, │ └─────────────────────┘ │ +│ │ sippy query, │ │ +│ │ post report) │ │ +│ └───────────────┘ │ +└─────────────────────────────────────────────────────────────────┘ + │ │ │ + ▼ ▼ ▼ + ┌─────────┐ ┌──────────────┐ ┌──────────────┐ + │ GitHub │ │ Claude API │ │ Sippy API │ + │ API │ │ (Vertex AI) │ │ (flake │ + │ (PRs, │ │ (Phase 2+ │ │ history) │ + │ checks,│ │ only) │ │ │ + │ comment)│ │ │ │ │ + └─────────┘ └──────────────┘ └──────────────┘ +``` + +--- + +## Configuration + + +| Variable | Default | Description | +| --------------------- | ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | +| `BOT_USER` | `openshift-app-platform-shift-bot` | Git identity used for bot commits (matches the GitHub App). Also used to detect bot's own replies in review threads (skip re-responding to self). | +| `PR_AGENT_MAX_PRS` | `4` | Maximum PRs to process per periodic run (kept low to stay within the Prow job timeout) | +| `MAX_BUDGET_PER_PR` | `5.00` | Maximum dollar amount to spend on Claude API per PR (passed to `--max-budget-usd`) | +| `MAX_COMMITS_PER_RUN` | `10` | Maximum total commits across all PRs in a single run | +| `MAX_COMMITS_PER_PR` | `3` | Maximum commits per individual PR processing | +| `MAX_DIFF_LINES` | `500` | Maximum lines changed by a single auto-fix before aborting | +| `PR_TIMEOUT_SECONDS` | `720` | Maximum seconds to spend processing a single PR (12 min). On timeout, posts partial report and continues. | +| `DRY_RUN` | `false` | When `true`, skips all file modifications, commits, and pushes | +| `SKIP_USERS` | `openshift-ci,openshift-bot,dependabot,codecov,sonarcloud,coderabbitai[bot]` | Comma-separated list of users whose comments are skipped | +| `RATE_LIMIT_SECONDS` | `60` | Delay between processing PRs in periodic mode | +| `RETEST_INFRA_FLAKES` | `false` | (Phase 2) When `true`, posts targeted `/test ` for infrastructure flakes. Max 2 retests per job per run. | +| `GCSWEB_BASE_URL` | `https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com` | Base URL for OpenShift CI gcsweb (Prow log fetching). Update if CI infrastructure migrates. | + + +### Required Prow Secrets + + +| Secret (in `test-credentials` namespace) | Mount Path | Purpose | +| -------------------------------------------------- | ------------------------- | -------------------------------------------------------------------------------------------- | +| `oap-lts-claude-gcp-vertex-sa` | `/var/run/gcloud-adc/` | GCP ADC JSON for Vertex AI Claude access (`application_default_credentials.json`) | +| `openshift-app-platform-shift-github-bot` | `/var/run/github-app/` | GitHub App ID (`app-id` key) and private key PEM (`private-key.pem` key) for generating installation tokens | + + +--- + +## Limitations + +- **AI may produce incorrect or incomplete solutions** — all fixes pushed by the agent must be reviewed by repository OWNERS before merging. +- **Complex issues may not be fully addressed** — multi-faceted build errors, test failures, and architectural issues require human intervention. +- **Rate limited**: 4 PRs per periodic run (configurable via `PR_AGENT_MAX_PRS`), 100 agentic turns per PR. +- **Cannot access private resources** — no access to internal systems beyond GitHub and Jira. +- **Cannot execute destructive operations** — no ability to force-push, rebase, or delete branches. Enforced via `--allowedTools` restrictions on Claude CLI invocations. +- **Concurrent processing race** — multiple Prow presubmit runs for the same PR (e.g., after rapid pushes) could process simultaneously. The consequence is duplicate work (not data loss): both runs may analyze the same failures and attempt the same fixes, with the second push either succeeding (identical fix) or gracefully failing (conflict detected by `git pull --rebase`). State persistence uses last-writer-wins, which may cause already-addressed comments to be re-analyzed on the next run. +- **Prow job timeout** — presubmit timeout is `2h30m0s`, providing ample time for CI polling plus analysis. The GitHub App installation token generated at job start is valid for 1 hour, which is sufficient for single-PR presubmit processing. +- **No periodic sweep** — Phase 1 is purely presubmit-driven. There is no periodic scanner catching PRs that were missed. If the presubmit is not configured for a repo, no monitoring occurs for that repo's PRs. +- **Cost** — deterministic classification handles ~80-90% of cases without Claude API cost. Claude Code is invoked only for `unknown` failures and review comment handling. +- **Target repo ci-operator config** — the `oape-ci-monitor` presubmit must be added to each target repo's ci-operator config in `openshift/release`. Requires a PR to `openshift/release` approved by the repo's CI admins. + +--- + +## Monitoring and Effectiveness + +### Performance Monitoring + +- **Prow job logs**: View at `https://prow.ci.openshift.org` → search for `oape-ci-monitor` job for the target repo. +- **GCS artifacts**: Build logs and artifacts stored in GCS buckets accessible via gcsweb (e.g., `gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com`). +- **PR comments**: The CI monitor report is posted directly to the PR, providing immediate visibility without navigating CI systems. +- Track job success/failure rates via Prow job history. + +### Metrics and Indicators + + +| Metric | Description | +| ------------------------ | ---------------------------------------------------- | +| PRs processed per run | Number of PRs successfully analyzed per periodic run | +| Auto-fixes applied | Count of trivial CI failures automatically resolved | +| Review threads addressed | Count of review comments handled by the agent | +| Fix success rate | Percentage of auto-fixes that pass subsequent CI | +| Time to CI-green | Duration from PR creation to all checks passing | + + +### Periodic Review Process + +The OAPE team should conduct monthly reviews: + +- Review auto-fix commits for quality and correctness. +- Track false positives (agent applied a fix that was wrong) and false negatives (agent missed a trivial fix). +- Adjust classification heuristics in the `ci-failure-analysis` skill based on results. +- Monitor Claude API costs and adjust `MAX_BUDGET_PER_PR` if needed. +- Review safety guardrail effectiveness — are blocked patterns correct? Are commit limits appropriate? + +--- + +## Summary + + +| # | Subtask | Type | Effort Estimate | +| --- | --------------------------------------------------------------------------------------------------------------- | ---------------------- | --------------- | +| 0 | Prow presubmit infrastructure + ci-operator config | Prow ci-operator Config | Medium | +| 1 | Create entrypoint script with PR discovery, prechecks, merge conflict detection, skip label, and state tracking | Script | Medium | +| 2 | Implement CI check monitoring via `gh pr checks` | Script | Medium | +| 3 | Implement CI failure log analysis with deterministic classification + Claude fallback | Script + Skill | Large | +| 4 | Implement trivial auto-fix engine with blobless clone, pre/post blocklist, global commit counter | Script | Large | +| 5 | Implement review comment monitoring and response with `--allowedTools` restrictions | Script + Claude Code | Medium | +| 6 | Wire together the PR processing pipeline (standalone scripts) | Script | Small | +| 7 | Implement safety guardrails and file-modification boundaries (sourced utility library, no dependencies) | Script + Skill | Medium | +| 8 | Implement status reporting with merge conflict section and PR comment summary | Script | Medium | +| 9 | PR agent testing and validation | Script | Small | +| 10 | Create `/oape:pr-agent` command | Command | Small | + + +### Files Created + + +| File | Purpose | +| -------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- | +| `docs/prow-ci-operator-config.yaml` | Reference ci-operator config for adding `oape-ci-monitor` presubmit to target repos in `openshift/release` | +| `scripts/pr-agent/entrypoint.sh` | Main orchestration script (both modes, merge conflict check, skip label filter, state tracking) | +| `scripts/pr-agent/ci-monitor.sh` | CI check fetching via `gh pr checks` and status aggregation (standalone executable) | +| `scripts/pr-agent/log-analyzer.sh` | Log fetching + deterministic classification + Claude fallback for unknowns (standalone executable) | +| `scripts/pr-agent/auto-fix.sh` | Trivial fix application with blobless clone, pre/post blocklist checks, global commit counter (standalone executable) | +| `scripts/pr-agent/review-handler.sh` | Review comment analysis and Claude Code invocation with `--allowedTools` restrictions (standalone executable) | +| `scripts/pr-agent/safety.sh` | Blocklists (category-aware go.sum exception), commit limits, audit logging, retry helper (sourced utility library) | +| `scripts/pr-agent/report.sh` | Status report generation with merge conflict section and PR comment posting (standalone executable) | +| `scripts/pr-agent/test-dry-run.sh` | Dry-run integration test script | +| `plugins/oape/skills/ci-failure-analysis/SKILL.md` | Claude Code skill for failure classification (content included via `cat` in prompts) | +| `plugins/oape/skills/pr-agent-safety/SKILL.md` | Safety guardrails skill (content included via `cat` in prompts) | +| `plugins/oape/commands/pr-agent.md` | `/oape:pr-agent` command for interactive developer use (Prow presubmit uses `monitor.sh`/`dispatch.sh` directly) | + + +### User Guide + +#### Viewing Agent Output + +Track PRs processed by the agent: + +- **Prow job logs**: Navigate to Prow CI dashboard (`prow.ci.openshift.org`), filter by job name `oape-ci-monitor` +- **Agent comments**: Look for comments containing `` on PRs in configured repos +- **GCS artifacts**: Available via gcsweb for the `oape-ci-monitor` job run + +#### Triggering On-Demand + +The agent triggers automatically as a Prow presubmit on every PR push, or manually: + +1. **Automatic (primary)**: The `oape-ci-monitor` presubmit runs automatically on every PR push in configured repos +2. **Via Prow chatops**: Comment `/test oape-ci-monitor` on the PR + +#### Skipping a PR + +To exclude a PR from automated processing, add the `pr-agent:skip` label. The presubmit will skip PRs with this label. + +#### Reprocessing + +The agent maintains lightweight state across runs via the PR report comment (tracking which CI jobs have been analyzed and which review comments have been addressed). On each run, already-processed items are skipped to avoid duplicate work. To force a full reprocessing of a PR, delete the agent's report comment (containing ``) from the PR, then trigger another run via `/test oape-ci-monitor`. + +--- + +## Implementation Phasing + +The subtasks above describe the full target architecture. Implementation is phased to deliver value incrementally and validate the approach before investing in the full design. + +### Phase 1: MVP — Prow Presubmit CI Monitor (Report-Only) + +**Goal**: Prove the concept by adding a Prow presubmit job to target repos that monitors CI and reports failures. The presubmit runs alongside other CI jobs, polls until all other checks reach a terminal state, then classifies failures and posts a structured report. `dispatch.sh` then logs planned next-step actions (no-op in Phase 1, real invocations in Phase 2+). No auto-fix, no review comment handling, no Claude dependency. + +**Architecture**: Each target repo's ci-operator config in `openshift/release` gains three additions from `docs/prow-ci-operator-config.yaml`: (1) an inline `ci-monitor-agent` image build, (2) a promotion exclusion, and (3) an `oape-ci-monitor` presubmit test. The presubmit builds the container, generates a GitHub App token from the mounted PEM key, and runs the analysis scripts. + +``` +PR push → Prow triggers oape-ci-monitor presubmit + → Build ci-monitor-agent container (inline Dockerfile) + → Generate GitHub App token from mounted PEM + → monitor.sh: polls gh pr checks → collects GCS artifacts → classifies → Sippy → report → result JSON + → dispatch.sh: reads result JSON → logs planned actions (Phase 1) / invokes auto-fix, Claude, /retest (Phase 2+) +``` + +| File | Purpose | Maps to Subtasks | +| ------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | ----------------- | +| `scripts/ci-monitor/monitor.sh` | CI monitor: polls checks, collects GCS artifacts, classifies failures, queries Sippy, generates report, posts comment, writes result JSON | 2, 3 (partial) | +| `scripts/ci-monitor/dispatch.sh` | Failure dispatch: reads result JSON, logs planned actions (Phase 1), invokes further oape-ai-e2e tools on failure (Phase 2+) | 6 (partial) | +| `docs/prow-ci-operator-config.yaml` | Reference ci-operator config template for adding `oape-ci-monitor` presubmit to target repos in `openshift/release` | 0 (partial) | + +**Validation**: The pipeline has been validated end-to-end via Prow rehearsal on [openshift/release#80727](https://github.com/openshift/release/pull/80727). The rehearsal detects the `openshift/release` context, switches to a real open PR on `openshift/must-gather-operator`, and runs the full pipeline including posting the analysis comment — allowing validation without merging the release PR first. + +**Scope**: Report-only CI monitoring via a Prow presubmit in each target repo. First target repo: must-gather-operator. The presubmit runs as `always_run: true, optional: true` and polls until all other checks are terminal. Once complete, it runs `monitor.sh` which fetches `gh pr checks`, collects `build-log.txt` from GCS for failed Prow jobs, classifies failures into categories (`install-failure`, `test-failure`, `build-failure`, `lint-failure`, `infra-flake`, `unknown`), queries Sippy for flake history, and posts a structured markdown report on the PR. A machine-readable JSON result (`ci-monitor-result.json`) includes suggested trigger actions (retest, auto-fix-lint, investigate). `dispatch.sh` reads this result and logs planned actions — in Phase 1 these are no-ops, in Phase 2+ they become real invocations of oape-ai-e2e tools. + +**Phase 1 enhancements (from PR #60 analysis):** +- **Release repo discovery**: `monitor.sh` fetches the ci-operator config from `openshift/release` for the target repo/branch, providing authoritative job metadata (required/optional, cluster_profile, OCP release version). Falls back to name-based heuristics if unavailable. +- **Non-test context exclusions**: Filters out non-CI contexts (`tide`, `Mergeable`, `DCO`, `CodeRabbit`, `stale`, `sonarcloud`, `codecov`) that should never be counted as failures. +- **Expanded failure patterns**: Infra-flake detection includes `registry.ci.openshift.org` errors, `etcdserver` timeouts, lease failures, cloud quota errors, `dial tcp` timeouts. Install-failure detection includes `level=fatal.*installer`, `bootstrapComplete` waits. +- **Dynamic Sippy release version**: Resolves OCP version from ci-operator config (`releases.latest.release.version`) or Prow job name pattern, providing accurate flake data per release. +- **Prow Job Breakdown table**: Report includes a table of ALL checks (pass/fail) with state, category, required/optional status, flake%, and recommended action. + +**Retained for future phases**: The PR agent scripts (`scripts/pr-agent/entrypoint.sh`, `safety.sh`) are retained as the foundation for `dispatch.sh` to invoke in Phase 2+. Since the `ci-monitor-agent` container includes all oape-ai-e2e scripts and plugins at build time, all tools are available at runtime. + +### Phase 2: Auto-Fix + Claude Intelligence + +**Goal**: Add CI-triggered auto-fix for trivial failures, Claude-powered analysis for unknown failures, and auto-retest for confirmed flakes. The CI monitor's `trigger_actions` output from Phase 1 drives dispatch. Incorporate context-aware analysis patterns from PR #60's ci-monitor skill. + +| File | Purpose | Maps to Subtasks | +| -------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | ---------------- | +| `scripts/pr-agent/auto-fix.sh` | Extracted auto-fix engine: `go fmt`, `goimports`, `make generate`, scoped to PR-changed files | 4 | +| `scripts/pr-agent/log-analyzer.sh` | Deterministic + Claude fallback classification for `unknown` failures | 3 | +| `plugins/oape/skills/ci-failure-analysis/SKILL.md` | Claude skill for unknown failure classification (reference: PR #60's `plugins/oape/skills/ci-monitor/SKILL.md`) | 3 | +| `scripts/pr-agent/safety.sh` | Retained guardrails: blocklist, audit log, commit limits, diff size guard | 7 | + +**Scope adds**: Auto-fix for `lint-failure` and `build-failure` categories, auto-retest (`/retest`) for `infra-flake`, Claude Code CLI fallback for `unknown` failures. `dispatch.sh` (already invoked by the Prow presubmit after `monitor.sh`) reads `ci-monitor-result.json` and takes action — no separate dispatch workflow needed. + +**Learnings from PR #60 to incorporate in Phase 2:** + +- **Auto-retest protocol**: When ALL failures on a PR are infra-flake (Mode E), `dispatch.sh` posts `/retest` automatically (max 2 per session). Only triggers when every failure is infrastructure-related. Disable with `--no-auto-retest`. Each auto-retest is logged in the report with timestamp, affected contexts, and outcome. + +- **On-demand PR diff fetch**: When a build/test failure references a specific file, fetch the diff for that file only (`gh pr diff $PR -- $FILE`) to correlate the error with the actual code change. Only fetch for files in `PR_CHANGED_FILES` — if the error is in a file not changed by the PR, flag it as a dependency or generated-code issue. + +- **Error signature hashing**: Normalize error messages (strip timestamps, line numbers, hex addresses `0x[a-f0-9]+`, UUIDs `[a-f0-9-]{36}`, temp paths `/tmp/[^ ]+`), then SHA-256 hash. Track `context_name -> error_hash` per fix round. If >= 75% of failed contexts share the same hash as the previous round, the fix was ineffective — stop the fix loop. + +- **Root cause tracing protocol**: Step-by-step diagnostic decision tree for each failure: + 1. Does the error reference a specific file? Is it in `PR_CHANGED_FILES`? → PR likely introduced the issue. + 2. Is it about a missing tool, command, or image? → Check ci-operator config's `container.from` or step `from:` image. + 3. Is it about authentication/credentials? → Check ci-operator `credentials` entries (Vault-injected, declared in `openshift/release`). + 4. Is it transient/environmental (network, quota, lease)? → Recommend `/retest`. + 5. Is it about missing generated code (`zz_generated.deepcopy.go`, CRD YAML)? → Check if `_types.go` changed but generated files weren't updated. + 6. None of the above → Report with all available evidence, confidence: low. + Each step cites concrete artifacts (log line, file path, config entry). Output format: numbered trace steps, fix location, fix owner, confidence level. + +- **PR change context**: Fetch changed files list per PR (`gh pr view --json files`). Classify change types: API (`_types.go`), controller (`controller|reconcil*.go`), test (`_test.go`), CRD (`crd/*.yaml`), RBAC (`rbac*.yaml`). Used for error-to-file correlation and stage-aware summary. + +- **Operator repo context**: Detect operator framework from `go.mod` (`sigs.k8s.io/controller-runtime` vs `github.com/openshift/library-go`). Detect Makefile presence, test directories. Used for targeted fix suggestions and local verification commands. + +- **Step registry resolution**: For failed Prow jobs with multi-stage steps, resolve step refs from `openshift/release` step registry (`ci-operator/step-registry/`). Maps "e2e-aws failed" to "step `openshift-e2e-test` failed, running `openshift-tests run openshift/conformance/parallel`". Resolved on demand only for failed jobs (saves API calls). + +- **Optional job severity**: Jobs marked `optional: true` in ci-operator config should never be labeled as "Blocker" or "Critical". Label as "Non-blocking (optional)" regardless of failure mode. Optional job failures should not change the PR's overall verdict from PASS to FAIL. + +### Phase 3: Review Comments + Full Design + +**Goal**: Complete the target architecture with review comment handling, the `/oape:pr-agent` command, and rollout to all target repos. + +| File | Purpose | Maps to Subtasks | +| ---------------------------------------------- | ------------------------------------------------------------------------------------------ | ---------------- | +| `scripts/pr-agent/review-handler.sh` | Review comment monitoring/response with restricted `--allowedTools` | 5 | +| `scripts/pr-agent/report.sh` | Extracted reporting logic (unified for CI monitor + PR agent) | 8 | +| `plugins/oape/skills/pr-agent-safety/SKILL.md` | Safety rules skill for Claude | 7 | +| `plugins/oape/commands/pr-agent.md` | `/oape:pr-agent` command for interactive + headless use | 10 | + +**Scope adds**: Review comment handling, `/oape:pr-agent` command, rollout of `oape-ci-monitor` presubmit to all repos in `team-repos.csv` (by adding ci-operator config snippets to each repo's config in `openshift/release`), full test suite. + +--- + +## Support and Feedback + +- **Slack channel**: #oape-support +- **Feedback**: File issues with label `pr-agent-feedback` +- **Urgent issues**: Contact OAPE team directly + diff --git a/README.md b/README.md index 758c3e6..79fed02 100644 --- a/README.md +++ b/README.md @@ -64,7 +64,7 @@ The following tools must be installed and available on your system: - **GitHub CLI (`gh`)**: [cli.github.com](https://cli.github.com/) - **make**: eg. `make generate`, `make build`, etc. -The container image used in [Dockerfile](./Dockerfile) already adds these dependencies, but you need to manage the credentials inside the container. +The container images in [images/](./images/) already add these dependencies, but you need to manage the credentials inside the container. ### Optional @@ -113,16 +113,16 @@ ln -s oape-ai-e2e ~/.cursor/commands/oape-ai-e2e | Plugin | Description | Commands | | ------------------------- | ---------------------------------------------- | --------------------------------------------------------------------------- | -| **[oape](plugins/oape/)** | AI-driven OpenShift operator development tools | `/oape:init`, `/oape:api-generate`, `/oape:api-generate-tests`, `/oape:api-implement`, `/oape:analyze-rfe`, `/oape:e2e-generate`, `/oape:predict-regressions`, `/oape:review`, `/oape:implement-review-fixes` | +| **[oape](plugins/oape/)** | AI-driven OpenShift operator development tools | `/oape:init`, `/oape:api-generate`, `/oape:api-generate-tests`, `/oape:api-implement`, `/oape:analyze-rfe`, `/oape:e2e-generate`, `/oape:predict-regressions`, `/oape:review`, `/oape:implement-review-fixes`, `/oape:pr-agent` | ## Commands ### `/oape:init` -- Clone an Operator Repository -Clones an allowed OpenShift operator repository by short name into the current directory. +Clones an allowed OpenShift operator repository into the current directory and checks out the specified base branch. ```shell -/oape:init cert-manager-operator +/oape:init https://github.com/openshift/cert-manager-operator main ``` ### `/oape:api-generate` -- Generate API Types from Enhancement Proposal @@ -189,13 +189,23 @@ Performs a production-grade code review that verifies code changes against Jira Automatically applies code fixes from a review report. ```shell -/oape:implement-review-fixes +/oape:implement-review-fixes +``` + +### `/oape:pr-agent` -- Monitor PR CI Status and Triage Failures + +Monitors a PR's CI checks, classifies failures deterministically, and posts a structured Markdown report as a PR comment. + +```shell +/oape:pr-agent https://github.com/openshift/cert-manager-operator/pull/123 +/oape:pr-agent https://github.com/openshift/cert-manager-operator/pull/123 --dry-run +/oape:pr-agent https://github.com/openshift/cert-manager-operator/pull/123 --monitor-only ``` **Typical workflow:** ```shell # Step 1: Clone the operator repository -/oape:init cert-manager-operator +/oape:init https://github.com/openshift/cert-manager-operator main # Step 2: Generate API types /oape:api-generate https://github.com/openshift/enhancements/pull/1234 @@ -219,10 +229,12 @@ Automatically applies code fixes from a review report. ```shell podman build -t quay.io/your-username/oape-ai:agent-worker -f images/agent-worker.Dockerfile . +podman build -t quay.io/your-username/oape-ai:ci-monitor -f images/ci-monitor.Dockerfile . podman build -t quay.io/your-username/oape-ai:gh-token-minter -f images/gh-token-minter.Dockerfile . podman build -t quay.io/your-username/oape-ai:go-server -f images/go-server.Dockerfile . podman push quay.io/your-username/oape-ai:agent-worker +podman push quay.io/your-username/oape-ai:ci-monitor podman push quay.io/your-username/oape-ai:gh-token-minter podman push quay.io/your-username/oape-ai:go-server ``` diff --git a/docs/prow-ci-operator-config.yaml b/docs/prow-ci-operator-config.yaml new file mode 100644 index 0000000..1938db0 --- /dev/null +++ b/docs/prow-ci-operator-config.yaml @@ -0,0 +1,177 @@ + # oape-ci-monitor — Prow presubmit job for CI failure analysis. +# +# Add the snippets below to the target repo's ci-operator config in +# openshift/release. Replace REPO_ORG, REPO_NAME, and BRANCH as needed. +# +# File: ci-operator/config/REPO_ORG/REPO_NAME/REPO_ORG-REPO_NAME-BRANCH.yaml +# +# The job runs as an optional, always-run presubmit alongside other CI jobs. +# It polls until all other checks reach a terminal state, then: +# 1. Collects Prow build logs from GCS via gcsweb +# 2. Classifies failures deterministically (regex-based, zero API cost) +# 3. Queries Sippy for flake history +# 4. Posts a structured analysis report as a PR comment +# +# Trigger manually: /test oape-ci-monitor +# +# Prerequisites: +# Secrets in test-credentials namespace: +# - oap-lts-claude-gcp-vertex-sa (GCP ADC for Vertex AI) +# - openshift-app-platform-shift-github-bot (GitHub App ID + private key) +# +# Auth strategy: +# Phase 1 (report-only): Tries GitHub App token first. If the App is not +# installed on the target repo, falls back to GITHUB_TOKEN for read + comment. +# Phase 2+ (auto-fix): Requires the GitHub App token for pushes that trigger CI +# (GITHUB_TOKEN pushes don't trigger downstream CI due to anti-recursion). + +# ───────────────────────────────────────────────────────────────────── +# 1. Add inline image build under images.items[] +# ───────────────────────────────────────────────────────────────────── +# +# images: +# items: +# - dockerfile_literal: |- +# FROM registry.access.redhat.com/ubi9/go-toolset +# USER 0 +# RUN dnf install -y git make jq && \ +# dnf install -y 'dnf-command(config-manager)' && \ +# dnf config-manager --add-repo https://cli.github.com/packages/rpm/gh-cli.repo && \ +# dnf install -y gh && \ +# dnf clean all +# WORKDIR /app +# RUN git clone --depth 1 -b OAPE-752 https://github.com/openshift-eng/oape-ai-e2e.git /tmp/oape && \ +# cp -r /tmp/oape/scripts /app/scripts && \ +# cp -r /tmp/oape/plugins /plugins && \ +# mkdir -p /config && cp -r /tmp/oape/deploy/config/* /config/ && \ +# rm -rf /tmp/oape +# RUN go install golang.org/x/tools/cmd/goimports@latest && \ +# curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b /usr/local/bin +# RUN dnf install -y nodejs npm && dnf clean all +# RUN git config --global user.name "openshift-app-platform-shift-bot" && \ +# git config --global user.email "267347085+openshift-app-platform-shift-bot@users.noreply.github.com" +# RUN chmod -R g=u /opt/app-root/src +# USER 1001 +# to: ci-monitor-agent + +# ───────────────────────────────────────────────────────────────────── +# 2. Exclude from promotion (add under promotion.to[].excluded_images) +# ───────────────────────────────────────────────────────────────────── +# +# promotion: +# to: +# - excluded_images: +# - ci-monitor-agent +# name: "5.0" +# namespace: ocp + +# ───────────────────────────────────────────────────────────────────── +# 3. Add presubmit test under tests[] +# ───────────────────────────────────────────────────────────────────── +- always_run: true + as: oape-ci-monitor + optional: true + steps: + test: + - as: monitor + commands: | + set -euo pipefail + + echo "[setup] Starting oape-ci-monitor for ${REPO_OWNER}/${REPO_NAME} PR#${PULL_NUMBER}" + + # --- Rehearsal detection --- + # Prow rehearsal runs against openshift/release, not the target repo. + # Switch to a real target-repo PR to validate the full pipeline. + if [[ "${REPO_NAME}" == "release" && "${REPO_OWNER}" == "openshift" ]]; then + echo "[setup] Detected openshift/release context — switching to test target" + export REPO_OWNER="REPO_ORG" + export REPO_NAME="REPO_NAME" + TEST_PR=$(curl -s "https://api.github.com/repos/${REPO_OWNER}/${REPO_NAME}/pulls?state=open&per_page=1" \ + | python3 -c "import sys,json; data=json.load(sys.stdin); print(data[0]['number'] if data else '')" 2>/dev/null || echo "") + if [[ -z "$TEST_PR" ]]; then + echo "[setup] No open PRs found on ${REPO_OWNER}/${REPO_NAME} — skipping" + exit 0 + fi + export PULL_NUMBER="$TEST_PR" + export PR_URL="https://github.com/${REPO_OWNER}/${REPO_NAME}/pull/${PULL_NUMBER}" + echo "[setup] Testing against ${REPO_OWNER}/${REPO_NAME}#${PULL_NUMBER}" + fi + + # --- GitHub auth: try App token, fall back to GITHUB_TOKEN --- + # App token is preferred (required for Phase 2+ pushes that trigger CI). + # For Phase 1 (report-only), GITHUB_TOKEN is sufficient for read + comment. + USE_APP_TOKEN="false" + if [[ -f /var/run/github-app/app-id && -f /var/run/github-app/private-key.pem ]]; then + echo "[auth] Attempting GitHub App token..." + APP_ID=$(cat /var/run/github-app/app-id) + PEM_PATH="/var/run/github-app/private-key.pem" + HEADER=$(printf '{"alg":"RS256","typ":"JWT"}' | openssl base64 -e -A | tr '+/' '-_' | tr -d '=') + NOW=$(date +%s); EXP=$((NOW + 300)) + PAYLOAD=$(printf '{"iat":%d,"exp":%d,"iss":"%s"}' "$NOW" "$EXP" "$APP_ID" | openssl base64 -e -A | tr '+/' '-_' | tr -d '=') + SIGNATURE=$(printf '%s' "${HEADER}.${PAYLOAD}" | openssl dgst -sha256 -sign "$PEM_PATH" -binary | openssl base64 -e -A | tr '+/' '-_' | tr -d '=') + JWT="${HEADER}.${PAYLOAD}.${SIGNATURE}" + + INSTALL_RESPONSE=$(curl -s -w "\n%{http_code}" -H "Authorization: Bearer ${JWT}" -H "Accept: application/vnd.github+json" \ + "https://api.github.com/repos/${REPO_OWNER}/${REPO_NAME}/installation") + HTTP_CODE=$(echo "$INSTALL_RESPONSE" | tail -1) + INSTALL_BODY=$(echo "$INSTALL_RESPONSE" | sed '$d') + + if [[ "$HTTP_CODE" -eq 200 ]]; then + INST_ID=$(echo "$INSTALL_BODY" | python3 -c "import sys,json; print(json.load(sys.stdin)['id'])") + TOKEN_RESPONSE=$(curl -s -w "\n%{http_code}" -X POST -H "Authorization: Bearer ${JWT}" -H "Accept: application/vnd.github+json" \ + "https://api.github.com/app/installations/${INST_ID}/access_tokens") + T_CODE=$(echo "$TOKEN_RESPONSE" | tail -1) + T_BODY=$(echo "$TOKEN_RESPONSE" | sed '$d') + if [[ "$T_CODE" -eq 201 ]]; then + export GH_TOKEN=$(echo "$T_BODY" | python3 -c "import sys,json; print(json.load(sys.stdin)['token'])") + USE_APP_TOKEN="true" + echo "[auth] GitHub App token generated successfully" + else + echo "[auth] WARN: App token creation failed (HTTP ${T_CODE}), falling back to GITHUB_TOKEN" + fi + else + echo "[auth] WARN: App not installed on ${REPO_OWNER}/${REPO_NAME} (HTTP ${HTTP_CODE}), falling back to GITHUB_TOKEN" + fi + else + echo "[auth] GitHub App credentials not mounted, using GITHUB_TOKEN" + fi + + if [[ "$USE_APP_TOKEN" != "true" ]]; then + if [[ -z "${GH_TOKEN:-}" && -z "${GITHUB_TOKEN:-}" ]]; then + echo "[auth] ERROR: No GitHub token available (App token failed and GITHUB_TOKEN not set)" >&2 + exit 1 + fi + export GH_TOKEN="${GH_TOKEN:-${GITHUB_TOKEN}}" + echo "[auth] Using GITHUB_TOKEN (Phase 1 report-only — sufficient for read + comment)" + echo "[auth] NOTE: Phase 2+ auto-fix pushes require the GitHub App to be installed on ${REPO_OWNER}/${REPO_NAME}" + fi + + # --- GCP auth for Claude (Vertex AI) fallback --- + export GOOGLE_APPLICATION_CREDENTIALS="/var/run/gcloud-adc/application_default_credentials.json" + export CLAUDE_CODE_USE_VERTEX="1" + export CLOUD_ML_REGION="global" + export ANTHROPIC_VERTEX_PROJECT_ID="itpc-gcp-hcm-pe-eng-claude" + + # --- Run CI monitor --- + export PR_URL="https://github.com/${REPO_OWNER}/${REPO_NAME}/pull/${PULL_NUMBER}" + export SKIP_POLL="false" + export SELF_JOB_NAME="oape-ci-monitor" + export BUILD_ID="${BUILD_ID:-}" + export OAPE_RUN_URL="${BUILD_LOG_URL:-}" + + gh auth setup-git + /app/scripts/ci-monitor/monitor.sh + /app/scripts/ci-monitor/dispatch.sh + credentials: + - mount_path: /var/run/gcloud-adc + name: oap-lts-claude-gcp-vertex-sa + namespace: test-credentials + - mount_path: /var/run/github-app + name: openshift-app-platform-shift-github-bot + namespace: test-credentials + from: ci-monitor-agent + resources: + requests: + cpu: "1" + memory: 500Mi + timeout: 2h30m0s diff --git a/go.work.sum b/go.work.sum index 02d75e2..16f8384 100644 --- a/go.work.sum +++ b/go.work.sum @@ -1,19 +1,44 @@ +cloud.google.com/go/compute/metadata v0.3.0 h1:Tz+eQXMEqDIKRsmY3cHTL6FVaynIjX2QxYC4trgAKZc= cloud.google.com/go/compute/metadata v0.3.0/go.mod h1:zFmK7XCadkQkj6TtorcaGlCW1hT1fIilQDwofLpJ20k= +github.com/NYTimes/gziphandler v0.0.0-20170623195520-56545f4a5d46 h1:lsxEuwrXEAokXB9qhlbKWPpo3KMLZQ5WB5WLQRW1uq0= github.com/NYTimes/gziphandler v0.0.0-20170623195520-56545f4a5d46/go.mod h1:3wb06e3pkSAbeQ52E9H9iFoQsEEwGN64994WTCIhntQ= +github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5 h1:0CwZNZbxp69SHPdPJAN/hZIm0C4OItdklCFmMRWYpio= github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5/go.mod h1:wHh0iHkYZB8zMSxRWpUBQtwG5a7fFgvEO+odwuTv2gs= +github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a h1:idn718Q4B6AGu/h5Sxe66HYVdqdGu2l9Iebqhi/AEoA= github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a/go.mod h1:lB+ZfQJz7igIIfQNfa7Ml4HSf2uFQQRzpGGRXenZAgY= +github.com/creack/pty v1.1.9 h1:uDmaGzcdjhF4i/plgjmEsriH11Y0o7RKapEf/LDaM3w= +github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da h1:oI5xCqsCo564l8iNU+DwB5epxmsaqB+rhGL0m5jtYqE= github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= +github.com/google/btree v1.0.1 h1:gK4Kx5IaGY9CD5sPJ36FHiBJ6ZXl0kilRiiCj+jdYp4= github.com/google/btree v1.0.1/go.mod h1:xXMiIv4Fb/0kKde4SpL7qlzvu5cMJDRkFDxJfI9uaxA= +github.com/gorilla/websocket v1.5.0 h1:PPwGk2jz7EePpoHN/+ClbZu8SPxiqlu12wZP/3sWmnc= github.com/gorilla/websocket v1.5.0/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE= +github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7 h1:pdN6V1QBWetyv/0+wjACpqVH+eVULgEjkurDLq3goeM= github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7/go.mod h1:FecbI9+v66THATjSRHfNgh1IVFe/9kFxbXtjV0ctIMA= +github.com/kisielk/errcheck v1.5.0 h1:e8esj/e4R+SAOwFwN+n3zr0nYeCyeweozKfO23MvHzY= +github.com/kisielk/gotool v1.0.0 h1:AV2c/EiW3KqPNT9ZKl07ehoAGi4C5/01Cfbblndcapg= +github.com/kr/pty v1.1.1 h1:VkoXIwSboBpnk99O/KFauAEILuNHv5DVFKZMBN/gUgw= +github.com/moby/spdystream v0.4.0 h1:Vy79D6mHeJJjiPdFEL2yku1kl0chZpJfZcPpb16BRl8= github.com/moby/spdystream v0.4.0/go.mod h1:xBAYlnt/ay+11ShkdFKNAG7LsyK/tmNBVvVOwrfMgdI= +github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f h1:y5//uYreIhSUg3J1GEMiLbxo1LJaP8RfCpH6pymGZus= github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f/go.mod h1:ZdcZmHo+o7JKHSa8/e818NopupXU1YMK5fe1lsApnBw= +github.com/peterbourgon/diskv v2.0.1+incompatible h1:UBdAOUP5p4RWqPBg048CAvpKN+vxiaj6gdUUzhl4XmI= github.com/peterbourgon/diskv v2.0.1+incompatible/go.mod h1:uqqh8zWWbv1HBMNONnaR/tNboyR3/BZd58JJSHlUSCU= +github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= +github.com/stretchr/objx v0.5.0 h1:1zr/of2m5FGMsad5YfcqgdqdWrIhu+EBEJRhR1U7z/c= +github.com/yuin/goldmark v1.2.1 h1:ruQGxdhGHe7FWOJPT0mKs5+pD2Xs1Bm/kdGlHO04FmM= +go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto= go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE= +golang.org/x/crypto v0.24.0 h1:mnl8DM0o513X8fdIkmyFE/5hTYxbwYOjDS/+rK6qpRI= golang.org/x/crypto v0.24.0/go.mod h1:Z1PMYSOR5nyMcyAVAIQSKCDwalqy85Aqn1x3Ws4L5DM= +golang.org/x/mod v0.17.0 h1:zY54UmvipHiNd+pm+m0x9KhZ9hl1/7QNMyxXbc6ICqA= golang.org/x/mod v0.17.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c= +golang.org/x/sync v0.7.0 h1:YsImfSBoP9QPYL0xyKJPq0gcaJdG3rInoqxTWbfQu9M= golang.org/x/sync v0.7.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk= +golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 h1:H2TDz8ibqkAF6YGhCdN3jS9O0/s90v0rJh3X/OLHEUk= golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2/go.mod h1:K8+ghG5WaK9qNqU5K3HdILfMLy1f3aNYFI/wnl100a8= +gopkg.in/evanphx/json-patch.v4 v4.12.0 h1:n6jtcsulIzXPJaxegRbvFNNrZDjbij7ny3gmSPG+6V4= gopkg.in/evanphx/json-patch.v4 v4.12.0/go.mod h1:p8EYWUEYMpynmqDbY58zCKCFZw8pRWMG4EsWvDvM72M= +k8s.io/gengo/v2 v2.0.0-20240228010128-51d4e06bde70 h1:NGrVE502P0s0/1hudf8zjgwki1X/TByhmAoILTarmzo= k8s.io/gengo/v2 v2.0.0-20240228010128-51d4e06bde70/go.mod h1:VH3AT8AaQOqiGjMF9p0/IM1Dj+82ZwjfxUP1IxaHE+8= diff --git a/images/ci-monitor.Dockerfile b/images/ci-monitor.Dockerfile new file mode 100644 index 0000000..52b252f --- /dev/null +++ b/images/ci-monitor.Dockerfile @@ -0,0 +1,27 @@ +FROM registry.access.redhat.com/ubi9/go-toolset + +USER 0 +RUN dnf install -y git make jq && \ + dnf install -y 'dnf-command(config-manager)' && \ + dnf config-manager --add-repo https://cli.github.com/packages/rpm/gh-cli.repo && \ + dnf install -y gh && \ + dnf clean all + +WORKDIR /app + +RUN go install golang.org/x/tools/cmd/goimports@latest && \ + curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b /usr/local/bin + +RUN dnf install -y nodejs npm && dnf clean all + +COPY scripts/ci-monitor/ /app/scripts/ci-monitor/ +COPY scripts/pr-agent/ /app/scripts/pr-agent/ +COPY plugins /plugins +COPY deploy/config/ /config/ + +RUN git config --global user.name "openshift-app-platform-shift-bot" && \ + git config --global user.email "267347085+openshift-app-platform-shift-bot@users.noreply.github.com" + +RUN chmod -R g=u /opt/app-root/src + +USER 1001 diff --git a/plugins/oape/README.md b/plugins/oape/README.md index 1268457..44d7adc 100644 --- a/plugins/oape/README.md +++ b/plugins/oape/README.md @@ -93,7 +93,7 @@ Analyzes a Jira Request for Enhancement (RFE) and generates a structured breakdo **Typical Workflow:** ```shell # Clone the operator repository (if not already cloned) -/oape:init cert-manager-operator +/oape:init https://github.com/openshift/cert-manager-operator main # Generate the API types /oape:api-generate https://github.com/openshift/enhancements/pull/1234 @@ -193,6 +193,48 @@ Generates e2e test artifacts for any OpenShift operator repository by discoverin See [e2e-test-generator/](e2e-test-generator/) for fixture templates and pattern documentation. +--- + +### `/oape:implement-review-fixes` + +Automatically applies code fixes from a review report produced by `/oape:review`, prioritized by severity. + +**Usage:** +```shell +/oape:implement-review-fixes +``` + +**What it does:** +1. **Parse** -- Extracts the `issues` array from the review report JSON. +2. **Sort** -- Orders issues by severity (CRITICAL first). +3. **Apply** -- Applies each suggested fix to the codebase. +4. **Verify** -- Confirms the build still passes after each fix. +5. **Report** -- Summarizes which fixes were applied and which failed. + +This command is invoked automatically at the end of `/oape:review` when the report contains issues. It can also be run standalone. + +--- + +### `/oape:pr-agent` + +Monitors a PR's CI status, classifies failures deterministically, and generates a structured Markdown status report posted as a PR comment. + +**Usage:** +```shell +/oape:pr-agent https://github.com/openshift/cert-manager-operator/pull/123 +/oape:pr-agent https://github.com/openshift/cert-manager-operator/pull/123 --dry-run +/oape:pr-agent https://github.com/openshift/cert-manager-operator/pull/123 --monitor-only +``` + +**What it does:** +1. **Prechecks** -- Validates the PR URL format, required tools (`gh`), and GitHub authentication. +2. **Repo Validation** -- Confirms the PR's repository is in the allowed list (`deploy/config/team-repos.csv`). +3. **CI Monitoring** -- Runs `scripts/pr-agent/entrypoint.sh` which fetches CI check status via `gh pr checks`, collects failure logs from Prow GCS and GitHub Actions. +4. **Classification** -- Classifies failures deterministically via regex patterns (trivial-format, trivial-lint, trivial-import, trivial-generated-files, build-error, test-failure, infra-flake, unknown). +5. **Reporting** -- Generates a structured Markdown report with pass/fail summary, failure categories, and recommended actions. Posts as a PR comment (unless `--dry-run`). + +**Note:** This is the interactive/developer entry point. The Prow presubmit (`oape-ci-monitor`) invokes `monitor.sh` and `dispatch.sh` directly. + ## Prerequisites - **go** -- Go toolchain diff --git a/plugins/oape/commands/pr-agent.md b/plugins/oape/commands/pr-agent.md new file mode 100644 index 0000000..fa05bf8 --- /dev/null +++ b/plugins/oape/commands/pr-agent.md @@ -0,0 +1,145 @@ +--- +description: Monitor CI status on a PR, classify failures, and generate a structured status report (Phase 1 report-only) +argument-hint: [--dry-run] [--monitor-only] +--- + +## Name +oape:pr-agent + +## Synopsis +```shell +/oape:pr-agent [--dry-run] [--monitor-only] +``` + +## Description + +The `oape:pr-agent` command runs the PR Lifecycle Agent against a single pull request. It fetches CI check status, collects failure logs from Prow GCS and GitHub Actions, classifies failures deterministically (regex-based), and generates a structured Markdown status report. + +**Phase 1 behaviour:** report-only. The agent posts a PR comment summarising CI results but does not apply auto-fixes or respond to review comments. + +## Arguments + +- `$1` (`PR-URL`): Full GitHub PR URL, e.g. `https://github.com/openshift/cert-manager-operator/pull/123`. **Required.** +- `--dry-run`: Run the full analysis pipeline without posting any PR comments or making mutations. +- `--monitor-only`: Skip auto-fix even if failures are detected (default Phase 1 behaviour). + +## Implementation + +### Step 0: Parse Arguments + +Parse the user's input. The first positional argument is the PR URL. Remaining arguments are flags. + +```bash +PR_URL="$1" +DRY_RUN_FLAG="" +MONITOR_FLAG="--monitor-only" + +for arg in "${@:2}"; do + case "$arg" in + --dry-run) DRY_RUN_FLAG="--dry-run" ;; + --monitor-only) MONITOR_FLAG="--monitor-only" ;; + esac +done +``` + +### Step 1: Validate PR URL + +Verify the PR URL matches the expected format: +```bash +if [[ ! "$PR_URL" =~ ^https://github.com/[^/]+/[^/]+/pull/[0-9]+$ ]]; then + echo "ERROR: Invalid PR URL format. Expected: https://github.com///pull/" + exit 1 +fi +``` + +### Step 2: Validate Repo Allowlist + +Extract owner/repo from the URL and verify the repository is listed in `deploy/config/team-repos.csv`: +```bash +OWNER=$(echo "$PR_URL" | sed 's|https://github.com/||;s|/pull/.*||' | cut -d/ -f1) +REPO=$(echo "$PR_URL" | sed 's|https://github.com/||;s|/pull/.*||' | cut -d/ -f2) +REPO_ROOT=$(git rev-parse --show-toplevel 2>/dev/null || pwd) + +if ! grep -q "https://github.com/${OWNER}/${REPO}" "${REPO_ROOT}/deploy/config/team-repos.csv" 2>/dev/null; then + echo "ERROR: ${OWNER}/${REPO} is not in the allowed repos list (deploy/config/team-repos.csv)" + exit 1 +fi +``` + +### Step 3: Run the Agent + +Invoke `scripts/pr-agent/entrypoint.sh` in on-demand mode: + +```bash +export RUNNER_TEMP="${RUNNER_TEMP:-/tmp}" +export GH_TOKEN="${GH_TOKEN:-$(gh auth token 2>/dev/null || echo '')}" + +"${REPO_ROOT}/scripts/pr-agent/entrypoint.sh" \ + --mode on-demand \ + --pr-url "$PR_URL" \ + $MONITOR_FLAG \ + $DRY_RUN_FLAG +``` + +### Step 4: Display Results + +After the script completes, read and display the generated report: + +```bash +PR_NUMBER=$(echo "$PR_URL" | grep -oP '[0-9]+$') +REPORT_FILE="${RUNNER_TEMP}/pr-agent-report-${OWNER}-${REPO}-${PR_NUMBER}.md" + +if [[ -f "$REPORT_FILE" ]]; then + echo "" + echo "==========================================" + echo " PR Agent Report" + echo "==========================================" + cat "$REPORT_FILE" +else + echo "No report file generated. Check the output above for errors." +fi +``` + +Also display a summary of the failure analysis if available: + +```bash +ANALYSIS_FILE="${RUNNER_TEMP}/failure-analysis-${OWNER}-${REPO}-${PR_NUMBER}.json" +if [[ -f "$ANALYSIS_FILE" ]]; then + TOTAL=$(jq 'length' "$ANALYSIS_FILE" 2>/dev/null || echo 0) + TRIVIAL=$(jq '[.[] | select(.category | startswith("trivial-"))] | length' "$ANALYSIS_FILE" 2>/dev/null || echo 0) + FLAKES=$(jq '[.[] | select(.category == "infra-flake")] | length' "$ANALYSIS_FILE" 2>/dev/null || echo 0) + UNKNOWN=$(jq '[.[] | select(.category == "unknown")] | length' "$ANALYSIS_FILE" 2>/dev/null || echo 0) + + echo "" + echo "Failure breakdown: ${TOTAL} total — ${TRIVIAL} trivial-fixable, ${FLAKES} infra-flakes, ${UNKNOWN} unknown" +fi +``` + +### Step 5: Offer Scheduled Re-check (optional) + +Ask the user if they want to schedule a periodic re-check: + +> Would you like to schedule a re-check in 10 minutes? I can use CronCreate to poll the PR again after CI has had time to re-run. + +If the user accepts, create a one-shot cron job: +``` +CronCreate(cron: " *", recurring: false, + prompt: "/oape:pr-agent --monitor-only") +``` + +## Examples + +1. **Monitor a PR (default report-only mode)**: + ```shell + /oape:pr-agent https://github.com/openshift/cert-manager-operator/pull/456 + ``` + +2. **Dry-run analysis (no PR comment posted)**: + ```shell + /oape:pr-agent https://github.com/openshift/cert-manager-operator/pull/456 --dry-run + ``` + +3. **Explicit monitor-only mode**: + ```shell + /oape:pr-agent https://github.com/openshift/must-gather-operator/pull/123 --monitor-only + ``` diff --git a/plugins/oape/skills/ci-monitor/SKILL.md b/plugins/oape/skills/ci-monitor/SKILL.md new file mode 100644 index 0000000..44752b3 --- /dev/null +++ b/plugins/oape/skills/ci-monitor/SKILL.md @@ -0,0 +1,994 @@ +--- +name: CI Monitor +description: Monitor CI/Prow job status for OpenShift operator PRs with adaptive polling, context-aware failure analysis, and optional fix-push-rewatch loop +--- + +# CI Monitor Skill + +## Persona + +You are an **OpenShift CI/Prow monitoring specialist**. You monitor GitHub CI checks and Prow status contexts for pull requests, collect failure evidence, classify root causes, and optionally apply fixes. You think in terms of: + +- **Signal fidelity**: Distinguishing genuine failures from infra flakes, bot statuses, and transient errors +- **Adaptive efficiency**: Minimizing GitHub API calls while never missing a state transition +- **Prow internals**: ci-operator configs, step registry references, GCS artifact layouts, JUnit conventions +- **Failure triage**: Classifying failures by mode (install, test, build, lint, infra) and mapping them to actionable fixes +- **Flake awareness**: Cross-referencing test history via Sippy to distinguish regressions from known flakes +- **Fix safety**: Only pushing fixes when confidence is high, the branch is correct, and the error is deterministically identified + +You are thorough (collect all failures before reporting), evidence-based (every classification cites log lines or JUnit entries), and budget-conscious (adaptive polling saves API calls). + +--- + +## Release Repo Discovery + +Before polling begins, fetch the ci-operator configuration for the target repository from `openshift/release`. This provides authoritative job metadata that replaces name-based guessing. + +**Time budget**: This entire section (Steps 1-3) should complete in under **60 seconds**. If any step hangs or takes longer, skip it and fall back to name-based classification. Do NOT spend extended time parsing or re-fetching configs. + +### Step 1: Fetch ci-operator Config + +Fetch the config **exactly once** and save to a local temp file. Do NOT re-fetch from the API for subsequent parsing -- always read from the local file. + +```bash +ORG=$(echo "$REPO" | cut -d'/' -f1) +REPO_NAME=$(echo "$REPO" | cut -d'/' -f2) +BRANCH=$(gh pr view "$PR_NUMBER" --repo "$REPO" --json baseRefName --jq '.baseRefName') + +CI_OP_CONFIG_PATH="ci-operator/config/$ORG/$REPO_NAME" +CONFIG_FILE="$ORG-$REPO_NAME-$BRANCH.yaml" +LOCAL_CONFIG="/tmp/ci-monitor-${ORG}-${REPO_NAME}-${BRANCH}-$(date +%s).yaml" + +# Fetch ONCE using raw.githubusercontent.com (faster, no base64 decoding needed) +curl -sf "https://raw.githubusercontent.com/openshift/release/master/$CI_OP_CONFIG_PATH/$CONFIG_FILE" \ + -o "$LOCAL_CONFIG" 2>/dev/null + +if [ ! -s "$LOCAL_CONFIG" ]; then + # Try master if branch-specific config not found + CONFIG_FILE="$ORG-$REPO_NAME-master.yaml" + curl -sf "https://raw.githubusercontent.com/openshift/release/master/$CI_OP_CONFIG_PATH/$CONFIG_FILE" \ + -o "$LOCAL_CONFIG" 2>/dev/null +fi + +if [ ! -s "$LOCAL_CONFIG" ]; then + echo "WARNING: No ci-operator config found for $REPO. Using name-based job classification." + USE_RELEASE_CONTEXT=false +else + USE_RELEASE_CONTEXT=true + echo "Release config saved to $LOCAL_CONFIG" +fi +``` + +**IMPORTANT**: All subsequent parsing in Steps 2 and 3 MUST read from `$LOCAL_CONFIG`, NOT from the GitHub API. Do not call `gh api` or `curl` for the same config file again. + +### Step 2: Parse Job Manifest + +Parse the locally saved config file (`$LOCAL_CONFIG`) to extract the job manifest. Do NOT fetch from GitHub again. + +Extract from the ci-operator config YAML a **job manifest** map with each test entry's properties: + +| Field | Source in Config | Use | +|-------|-----------------|-----| +| `job_name` | `tests[].as` | Match against Prow context names | +| `job_type` | Derived: `cluster_profile` present = slow; `container` only = fast | Authoritative fast/slow classification | +| `always_run` | `tests[].always_run` | Required vs conditional job | +| `run_if_changed` | `tests[].run_if_changed` | Conditional trigger pattern | +| `optional` | `tests[].optional` | Can fail without blocking merge | +| `cluster_profile` | `tests[].steps.cluster_profile` | Cloud provider for infra-flake correlation | +| `release_version` | `releases.latest.release.version` or `releases.latest.release.channel` | Sippy query parameter (resolves `` placeholder) | +| `test_steps` | `tests[].steps.test[].ref` or `tests[].steps.test[].as` | Step registry references | +| `commands` | `tests[].commands` | Direct command string (container tests) | +| `credentials` | `tests[].steps.credentials[].name` and `.namespace` | Vault-injected secrets; trace auth failures here | + +### Step 3: Resolve Step Registry References + +For each `test_steps` reference, resolve to the actual commands using the step registry naming convention. + +The step registry maps ref names to directory paths by splitting on `-` at component boundaries. The directory path uses `/` separators, and files follow strict suffixes: + +| Component Type | Suffix | Example | +|---------------|--------|---------| +| Step ref | `-ref.yaml` | `openshift-e2e-test-ref.yaml` | +| Commands script | `-commands.sh` | `openshift-e2e-test-commands.sh` | +| Chain | `-chain.yaml` | `ipi-install-aws-chain.yaml` | +| Workflow | `-workflow.yaml` | `ipi-aws-workflow.yaml` | + +Resolution procedure: + +```bash +resolve_step_ref() { + local STEP_REF="$1" + # Convert ref name to directory path: openshift-e2e-test -> openshift/e2e/test + local STEP_DIR=$(echo "$STEP_REF" | sed 's|-|/|g') + local REG_BASE="ci-operator/step-registry" + + # Fetch the ref YAML + local REF_YAML=$(gh api "repos/openshift/release/contents/$REG_BASE/$STEP_DIR/${STEP_REF}-ref.yaml" \ + --jq '.content' 2>/dev/null | base64 -d 2>/dev/null || echo "") + + if [ -z "$REF_YAML" ]; then + echo "STEP_REF_UNRESOLVED" + return + fi + + # Extract image and commands file + local STEP_IMAGE=$(echo "$REF_YAML" | grep 'from:' | head -1 | awk '{print $2}') + local COMMANDS_FILE=$(echo "$REF_YAML" | grep 'commands:' | head -1 | awk '{print $2}') + + # Fetch the actual commands script + local COMMANDS_SCRIPT="" + if [ -n "$COMMANDS_FILE" ]; then + COMMANDS_SCRIPT=$(gh api "repos/openshift/release/contents/$REG_BASE/$STEP_DIR/$COMMANDS_FILE" \ + --jq '.content' 2>/dev/null | base64 -d 2>/dev/null || echo "") + fi + + echo "IMAGE=$STEP_IMAGE" + echo "COMMANDS_FILE=$COMMANDS_FILE" + echo "SCRIPT_PREVIEW=$(echo "$COMMANDS_SCRIPT" | head -20)" +} +``` + +Resolve step refs **on demand** after Phase 2 identifies failed jobs, not upfront for all jobs (saves API calls). + +### Graceful Fallback + +If `USE_RELEASE_CONTEXT=false`, all downstream phases fall back to name-based heuristics. The release context is an enrichment layer, not a hard dependency. + +--- + +## Adaptive Polling Algorithm + +### Multi-PR Polling Strategy + +When monitoring multiple PRs, use a **single shared polling loop** that processes all PRs in each cycle. Do NOT poll PRs sequentially (that would triple cycle time for 3 PRs). + +Each poll cycle: + +```text +1. For each active PR: + a. Check SHA (1 call) + b. Fetch statusCheckRollup (1 call) + c. Fetch commit statuses (1 call) + d. Update signals +2. Determine interval from combined context state across all PRs +3. Emit progress report +4. Sleep interval += 3 calls per active PR per cycle +``` + +**Combined interval selection**: The polling interval is determined by the **slowest active PR**. If any active PR has slow contexts pending, use the slow interval. If all active PRs have only fast contexts pending, use the fast interval. + +**Early PR completion**: When all contexts for a PR become terminal, mark that PR as complete and skip its 3 API calls in subsequent cycles. This saves budget as PRs finish at different times. + +```text +Example: monitoring PR #1, PR #2, PR #3 + Cycle 1-5: poll all 3 PRs (9 calls/cycle) + Cycle 6: PR #1 complete, skip it (6 calls/cycle) + Cycle 7-12: poll PR #2, PR #3 (6 calls/cycle) + Cycle 13: PR #2 complete (3 calls/cycle) + Cycle 14-20: poll PR #3 only (3 calls/cycle) +``` + +**SHA changes are per-PR**: A SHA change on PR #1 clears signals and resets the timer for PR #1 only. PR #2 and PR #3 continue unaffected. + +### Record Initial State + +```bash +for PR_NUMBER in "${PR_NUMBERS[@]}"; do + TRACKED_SHA[$PR_NUMBER]=$(gh pr view "$PR_NUMBER" --repo "$REPO" --json headRefOid --jq '.headRefOid') + SIGNALS[$PR_NUMBER]="{}" + PR_COMPLETE[$PR_NUMBER]=false +done +FIX_ATTEMPT=0 +ROUND_START=$(date +%s) +``` + +For each PR, maintain independently: +- `tracked_sha`: the HEAD SHA being monitored +- `signals`: map of context name to `{state, started_at, target_url, provider}` +- `pr_complete`: whether all contexts are terminal (skip in future cycles) + +### Job Classification + +When `USE_RELEASE_CONTEXT=true`, classify jobs from the ci-operator config: + +| Config Property | Classification | +|----------------|----------------| +| `container:` present, no `cluster_profile` | **Fast** | +| `cluster_profile:` present | **Slow** | +| `steps:` with workflow referencing `ipi-*` or `upi-*` | **Slow** | + +When `USE_RELEASE_CONTEXT=false`, fall back to the name-based pattern table: + +| Tier | Patterns | Classification | +|------|----------|----------------| +| Slow (checked first) | `e2e`, `install`, `cluster`, `conformance`, `upgrade`, `aws`, `gcp`, `azure`, `metal`, `vsphere`, `libvirt`, `ovirt`, `openstack` | Slow | +| Fast | `lint`, `vet`, `verify`, `verify-deps`, `unit`, `validate-boilerplate`, `coverage`, `go-build`, `images`, `bundle`, `shellcheck`, `gosec` | Fast | +| Fallback | Any context not matching either tier | Fast (conservative: poll at 60s to avoid missing short jobs) | + +Slow patterns are checked first. If a context name matches both tiers (e.g., `test-e2e-aws`), the slow match wins. + +### Interval Selection + +After the first successful poll, select the polling interval: + +| Condition | Interval | +|-----------|----------| +| Any fast context still pending | **60s** (fast phase) | +| Only slow contexts pending, running < 45 min | **120s** (slow phase) | +| Only slow contexts pending, running >= 45 min | **60s** (finishing phase) | +| All contexts terminal | Exit polling | + +### Auto-Adjusted Timeout + +After the first poll, auto-reduce timeout based on detected job types: + +| Detected Jobs | Timeout Cap | +|---------------|-------------| +| Only fast jobs (lint/verify/unit/build/images) | **30 min** | +| e2e without cluster install | **60 min** | +| e2e with cluster install | Full `--timeout-min` (default 120) | + +When `USE_RELEASE_CONTEXT=true`, this is authoritative (based on `cluster_profile` presence). When false, it is heuristic (based on name patterns). + +### SHA Change Detection + +On every poll iteration, for each active PR, before checking context states: + +```bash +CURRENT_SHA=$(gh pr view "$PR_NUMBER" --repo "$REPO" --json headRefOid --jq '.headRefOid') + +if [ "$CURRENT_SHA" != "${TRACKED_SHA[$PR_NUMBER]}" ]; then + echo "[ci-monitor] SHA changed on PR #$PR_NUMBER: ${TRACKED_SHA[$PR_NUMBER]} -> $CURRENT_SHA" + TRACKED_SHA[$PR_NUMBER]="$CURRENT_SHA" + # Clear accumulated results for THIS PR only — they are stale + SIGNALS[$PR_NUMBER]="{}" + # Mark PR as active again (in case it was previously complete) + PR_COMPLETE[$PR_NUMBER]=false + # Wait for Prow to register new contexts + sleep "$SHA_SETTLE_SEC" + # Reset round timer for this PR + ROUND_START=$(date +%s) + continue +fi +``` + +**Why 90s settle**: After a push, Prow needs 30-60s to register new status contexts. Polling immediately would see "0 pending, 0 failed" and incorrectly conclude everything passed. + +### Retest Detection (no SHA change) + +When `/retest` or `/test ` is commented, the SHA stays the same but Prow creates new job runs. Detect by comparing `started_at` timestamps: + +```bash +for CONTEXT_NAME in $(echo "$CURRENT_CONTEXTS" | jq -r '.[].name'); do + PREV_STATE=$(echo "$SIGNALS" | jq -r --arg n "$CONTEXT_NAME" '.[$n].state // empty') + PREV_STARTED=$(echo "$SIGNALS" | jq -r --arg n "$CONTEXT_NAME" '.[$n].started_at // empty') + CURR_STATE=$(echo "$CURRENT_CONTEXTS" | jq -r --arg n "$CONTEXT_NAME" \ + '.[] | select(.name==$n) | .state') + CURR_STARTED=$(echo "$CURRENT_CONTEXTS" | jq -r --arg n "$CONTEXT_NAME" \ + '.[] | select(.name==$n) | .started_at') + + if echo "failure success cancelled timed_out" | grep -qw "$PREV_STATE"; then + if echo "pending queued" | grep -qw "$CURR_STATE"; then + echo "Retest detected: $CONTEXT_NAME restarted (terminal -> pending)" + # Reset signal to pending + elif [ -n "$CURR_STARTED" ] && [ -n "$PREV_STARTED" ] && \ + [ "$CURR_STARTED" \> "$PREV_STARTED" ]; then + echo "Rerun detected: $CONTEXT_NAME completed a new run" + # Update signal with new result + fi + fi +done +``` + +### Unified Signal Tracking + +Merge both Checks API and commit status contexts into a single list. + +**How to fetch signals** (use these specific commands to avoid `gh` CLI field errors): + +```bash +# GitHub Checks (via GraphQL statusCheckRollup) — works reliably +gh pr view "$PR_NUMBER" --repo "$REPO" --json statusCheckRollup \ + --jq '.statusCheckRollup[] | "\(.name)\t\(.state)\t\(.detailsUrl)"' + +# Prow commit statuses (via REST API) — more reliable than gh pr checks +gh api "repos/$REPO/commits/$TRACKED_SHA/status" \ + --jq '.statuses[] | "\(.context)\t\(.state)\t\(.target_url)"' +``` + +Do NOT use `gh pr checks --json status,conclusion` -- the `status` field is not available in all `gh` versions and will error with "Unknown JSON field." + +For each entry track: +- `name` / `context` +- `state` (`queued`, `in_progress`, `pending`, `success`, `failure`, `cancelled`, `timed_out`, `action_required`, `skipped`) +- `provider` (`github-actions`, `github-check`, `prow-status-context`) +- `target_url` / `details_url` +- `started_at` + +### Non-Test Context Exclusions + +Exclude or handle non-CI contexts separately. Do not count these as failures: + +| Context Pattern | Classification | Action | +|-----------------|---------------|--------| +| `tide` | Merge gate | Report description (e.g., "needs: lgtm, approved"), do not treat as failure | +| `CodeRabbit` | Review bot | Ignore entirely | +| `Mergeable` | GitHub merge check | Report status, do not count as CI failure | +| `DCO` | Commit signing | Report status, do not count as CI failure | +| `stale` | Staleness bot | Ignore entirely | + +### User-Defined Context Exclusions (`--ignore-context`) + +When `--ignore-context ` is provided (one or more times), any CI context whose name contains the pattern (case-insensitive substring match) is treated the same as `CodeRabbit` or `stale` — **ignored entirely**. The context is: +- Excluded from the pending/running count (does not delay termination) +- Excluded from the failure count (does not affect the verdict) +- Not included in the Prow Job Breakdown table +- Not fetched for evidence collection + +This is primarily used when the ci-monitor runs as a CI job itself, to prevent it from watching its own check (e.g., `--ignore-context oape-ci-monitor`). + +When `USE_RELEASE_CONTEXT=true`, also check `tests[].optional: true` from the job manifest. Optional jobs that fail are reported but do not block the overall verdict. + +**Severity labeling for optional jobs**: If a job is marked `optional: true`, NEVER label its failure as "Blocker" or "Critical Severity." Label it as "Non-blocking (optional)" regardless of the failure mode. Optional job failures should not change the PR's overall verdict from PASS to FAIL. + +### Progress Reporting + +Emit status updates during polling so the user has visibility into long monitoring sessions. + +**Per-poll one-liner** (after every poll iteration): + +```text +[ci-monitor] 12:34 | PR #342 | 8/12 complete | 3 pending (e2e-aws, e2e-gcp, upgrade) | 1 failed (lint) | elapsed: 23m | next poll: 120s +``` + +**Milestone summary** (every 10 minutes): + +```text +[ci-monitor] === 30m checkpoint === + PR #342: 10/12 complete | 2 pending: e2e-aws (slow, ~15m est.), upgrade (slow, ~20m est.) + PR #343: 6/6 complete | ALL PASSED + PR #344: 4/8 complete | 1 failed: unit | 3 pending + API calls so far: 180 | Rate limit remaining: 4,820 + Auto-retests posted: 0/2 +``` + +**Event-driven reports** (immediately when detected): + +```text +[ci-monitor] SHA changed on PR #342 (abc1234 -> def5678). Clearing stale results, waiting 90s... +[ci-monitor] Retest detected: e2e-aws restarted on PR #342 (terminal -> pending) +[ci-monitor] Auto-retest posted on PR #344: all 2 failures are infra flakes. Retest 1/2. +[ci-monitor] PR #343 complete: ALL PASSED (6/6) +``` + +Track for milestone summaries: +- `POLL_COUNT`: total poll iterations +- `API_CALL_COUNT`: running total of GitHub API calls +- `LAST_MILESTONE`: timestamp of last milestone report (emit every 10 min) + +### Termination Conditions + +Stop polling a PR when: +1. No checks AND no status contexts are `pending`, `in_progress`, or `queued`. +2. Timeout reached -- mark unresolved signals as `timed_out`. +3. PR state changed to `CLOSED` or `MERGED`. + +--- + +## Evidence Collection Procedures + +For every signal in a terminal failure state (`failure`, `cancelled`, `timed_out`), gather evidence using the strategy appropriate to its provider. + +**CRITICAL RULES to prevent loops**: +- Fetch each artifact **exactly once**. Do NOT retry failed downloads. If a `curl` fails, mark as `partial` and move on. +- Do NOT re-fetch an artifact you already have. Before fetching, check if the local file already exists. +- **Time budget**: Evidence collection for ALL failed jobs combined should complete in under **3 minutes**. If it takes longer, stop collecting and proceed to classification with whatever evidence you have. +- Save all artifacts to a unique session directory: `/tmp/ci-monitor-$PR_NUMBER-$(date +%s)/` +- Process each failed context ONCE in a single pass -- do NOT loop back to re-process contexts. + +### GitHub Actions Failures + +Extract the run ID from the details URL and fetch failed logs: + +```bash +gh run view "$RUN_ID" --repo "$REPO" --log-failed +``` + +### Prow Status Context Failures + +For each failed `ci/prow/*` context: + +1. **Parse the Prow job URL** from `target_url`. Extract: + - GCS bucket path (e.g., `gs/test-platform-results/pr-logs/pull/.../`) + - Job name (e.g., `pull-ci-openshift-must-gather-operator-master-validate-boilerplate`) + - Build ID + +2. **Derive artifact base URL** from the `target_url`: + +```bash +ARTIFACT_BASE=$(echo "$TARGET_URL" | sed 's|https://prow.ci.openshift.org/view/gs/|https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/|') +``` + +3. **Download key artifacts in a single pass** (skip in `--fast` mode except `build-log.txt`): + +For each failed Prow context, run ONE batch of downloads. Do NOT loop or retry: + +```bash +ARTIFACT_DIR="/tmp/ci-monitor-$PR_NUMBER-$(date +%s)/$JOB_NAME" +mkdir -p "$ARTIFACT_DIR" + +# Download all artifacts in one pass — no retries +curl -sf "$ARTIFACT_BASE/build-log.txt" -o "$ARTIFACT_DIR/build-log.txt" 2>/dev/null || true +curl -sf "$ARTIFACT_BASE/finished.json" -o "$ARTIFACT_DIR/finished.json" 2>/dev/null || true + +# Skip these in --fast mode +if [ "$FAST_MODE" != true ]; then + curl -sf "$ARTIFACT_BASE/prowjob.json" -o "$ARTIFACT_DIR/prowjob.json" 2>/dev/null || true +fi + +# Mark which artifacts were obtained +echo "Artifacts: $(ls "$ARTIFACT_DIR" 2>/dev/null | tr '\n' ', ')" +``` + +| Artifact | Path | Purpose | +|----------|------|---------| +| `build-log.txt` | `$ARTIFACT_BASE/build-log.txt` | Primary CI log -- always fetch | +| `finished.json` | `$ARTIFACT_BASE/finished.json` | Exit code, timestamps, result | +| `prowjob.json` | `$ARTIFACT_BASE/prowjob.json` | Full Prow job spec (skip in fast mode) | +| `junit*.xml` | `$ARTIFACT_BASE/artifacts//junit*.xml` | Per-test pass/fail with error messages (skip in fast mode) | +4. **Parse JUnit XML** (unless `--fast`): + - Enumerate every `` with a `` or `` child. + - Extract: test name, class name, failure message, stack trace snippet (first 40 lines). + - Count total tests, passed, failed, skipped, errored. + +5. **Detect must-gather availability** (unless `--fast`): + - MUST check for jobs that provision clusters (Mode A install failures or Mode B e2e test failures with `cluster_profile` present in the job manifest). + - Skip for lint, unit, build, and container-only jobs -- they never produce must-gather artifacts. + - Path: `$ARTIFACT_BASE/artifacts/*/must-gather.tar` + - Check availability: + ```bash + MUST_GATHER_URL="$ARTIFACT_BASE/artifacts/must-gather/must-gather.tar" + if curl -sf --head "$MUST_GATHER_URL" >/dev/null 2>&1; then + echo "must-gather available: $MUST_GATHER_URL" + fi + ``` + - If present, include the download URL in the report's Evidence section. + - If absent, note "must-gather: not available" in the report. + +6. If any artifact fetch fails, mark evidence as `partial` and **move on immediately**. Do NOT retry. Do NOT re-fetch. The `|| true` in the download commands ensures failures are silent and non-blocking. + +### On-Demand PR Diff Fetch + +When a failure is identified and the failing file is in `PR_CHANGED_FILES`, fetch the diff for that specific file to correlate the error with the actual code change: + +```bash +FAILING_FILE="pkg/controller/foo.go" +if echo "$PR_CHANGED_FILES" | grep -qx "$FAILING_FILE"; then + FILE_DIFF=$(gh pr diff "$PR_NUMBER" --repo "$REPO" -- "$FAILING_FILE" 2>/dev/null || echo "") +fi +``` + +This is fetched **per failing file, on demand** -- not upfront for the entire PR. Include the relevant diff excerpt (max 30 lines around the error) in the report's Evidence section. + +### Step-Level Failure Mapping (when release context available) + +When `USE_RELEASE_CONTEXT=true` and the failed job uses multi-stage steps: + +1. Identify the failing step name from the build log (look for `step "" failed`). +2. Resolve the step ref using the step registry (see Release Repo Discovery, Step 3). +3. Record the step's image, commands file, and script preview in the evidence. + +This maps a generic "e2e-aws failed" to "step `openshift-e2e-test` failed, which runs `openshift-tests run openshift/conformance/parallel` in the `tests` image." + +--- + +## Failure Classification Rules + +For each failed job, classify into one of the following failure modes. Apply rules in order -- first match wins. + +### Mode A: Install Failure + +**Detection** (match ANY): +- JUnit contains a test matching `install should succeed:*` +- JUnit test names containing `cluster-install`, `bootstrap`, `infrastructure-setup`, or `infra-setup` +- Build log matches `level=fatal.*installer`, `cluster creation failed`, `bootstrap.*timed out`, or `waiting for bootstrapComplete` +- `finished.json` has `result: "ABORTED"` with install-stage timestamps +- When release context available: the failing step ref is part of a `pre:` chain in the workflow (install phase) + +**Analysis focus**: Identify install stage, extract installer errors. + +### Mode B: Test Failure (e2e, unit, integration) + +**Detection**: JUnit contains failed `` entries that are NOT install tests (Mode A). + +**Analysis focus**: List failed tests, classify isolation/co-failure/mass-failure patterns. When release context available, include the test step ref and its commands. + +### Mode C: Build / Compile Failure + +**Detection** (match ANY): +- Build log contains Go compile errors (`cannot find package`, `undefined:`, `syntax error`, `imported and not used`) +- Build log contains `make: *** [...] Error` with compile-related targets +- No JUnit output present (build failed before tests ran) + +**Analysis focus**: Extract compiler errors. When operator repo context available, map errors to local Go packages via `GO_MODULE`. When PR change context available, check if the failing file is in `PR_CHANGED_FILES` -- if yes, fetch the diff for that file on demand and correlate the error line with the actual code change: + +```bash +# Fetch diff only for the specific failing file +gh pr diff "$PR_NUMBER" --repo "$REPO" -- "$FAILING_FILE" +``` + +If the error is in a file NOT in `PR_CHANGED_FILES`, it may be a dependency or generated-code issue (e.g., `zz_generated.deepcopy.go` not regenerated after types changed). + +### Mode D: Lint / Static Analysis / Boilerplate + +**Detection** (match ANY): +- Job name contains `lint`, `vet`, `verify`, `validate-boilerplate`, `verify-deps` +- When release context available: job's `commands` field contains `make lint`, `make verify`, `make vet` + +**Analysis focus**: Extract lint/validation errors, distinguish CI image issues from actual drift. + +### Mode E: CI Infrastructure / Transient + +**Detection** (match ANY of these patterns in build log, with no test-level errors): +- `registry.ci.openshift.org.*timeout` or `registry.ci.openshift.org.*error` +- `ImagePullBackOff`, `ErrImagePull`, `ImagePullErr` +- `i/o timeout`, `connection refused`, `dial tcp.*timeout` +- `etcdserver: request timed out`, `lease lost` +- `error creating.*instance`, `quota exceeded`, `InsufficientInstanceCapacity` +- `unable to get lease`, `failed to acquire lease` +- `context deadline exceeded` with no test failures +- Error in Prow infrastructure (pod scheduling, volume mount) rather than in test code + +**Analysis focus**: Mark as `probable-infra-flake`. If auto-retest is enabled, post `/retest` automatically (see Auto-Retest Protocol below). Otherwise, recommend `/retest` in the report. + +--- + +## Auto-Retest Protocol + +When `--no-auto-retest` is NOT set and Mode E (infra flake) is detected, automatically post a `/retest` comment to re-trigger failed jobs. + +### Eligibility + +All conditions must be met: +- ALL failures on the PR are Mode E (infra flake). If any failure is Mode A/B/C/D, do not auto-retest -- fix the real failure first. +- The failing context is not optional (`tests[].optional != true` when release context available). +- `RETEST_COUNT < 2` for this monitoring session (hard cap to prevent infinite retest loops). + +### Procedure + +```bash +RETEST_COUNT=0 +MAX_AUTO_RETESTS=2 + +# After Phase 3 classifies all failures for a PR: +ALL_MODE_E=true +for FAILURE in $(pr_failures "$PR_NUMBER"); do + if [ "$(get_failure_mode "$FAILURE")" != "E" ]; then + ALL_MODE_E=false + break + fi +done + +if [ "$ALL_MODE_E" = true ] && [ "$RETEST_COUNT" -lt "$MAX_AUTO_RETESTS" ]; then + echo "[ci-monitor] All failures on PR #$PR_NUMBER are infra flakes. Posting /retest..." + gh pr comment "$PR_NUMBER" --repo "$REPO" --body "/retest" + RETEST_COUNT=$((RETEST_COUNT + 1)) + echo "[ci-monitor] Auto-retest $RETEST_COUNT/$MAX_AUTO_RETESTS posted. Resuming polling..." + # Polling loop naturally handles re-poll: contexts go back to pending +fi +``` + +### Guardrails + +- **Max 2 auto-retests** per monitoring session. After 2 retests, if infra flakes persist, report them and stop. +- **Only when ALL failures are Mode E**. A single real failure (Mode A/B/C/D) disables auto-retest for that PR. +- **Logged in final report**: each auto-retest is recorded with timestamp, affected contexts, and outcome. +- **Disabled with `--no-auto-retest`**: users can opt out entirely. + +### Interaction with Fix Loop + +Auto-retest and the fix loop serve different purposes and apply to different failure modes. Their ordering: + +1. Polling completes (all contexts terminal). +2. Evidence collection (Phase 2) runs for all failed contexts. +3. Failure classification (Phase 3) assigns a mode to each failure. +4. **Auto-retest evaluated first**: if ALL failures on a PR are Mode E, post `/retest` and resume polling. The fix loop is NOT entered. +5. **Fix loop evaluated second**: if any failures are Mode B/C/D (and auto-retest did not trigger), enter the fix loop. + +If auto-retest triggers and the retry produces a new real failure (Mode B/C/D), the next classification round will skip auto-retest (not all failures are Mode E anymore) and enter the fix loop instead. + +--- + +## Deep Analysis Protocol + +### Sippy Historical Pass Rate Lookup (MANDATORY for test failures) + +When any test failure (Mode B) is detected, you MUST query Sippy for historical pass rates. Do NOT skip this step. Do NOT report "Sippy queries: 0 (skipped)" when a release version is available. + +```bash +RELEASE_VERSION="$RESOLVED_RELEASE" +curl -sf "https://sippy.dptools.openshift.org/api/tests?release=$RELEASE_VERSION&filter.test_name=$TEST_NAME" +``` + +**How to resolve `RELEASE_VERSION`** (try in order): +1. From ci-operator config: `releases.latest.release.version` or `releases.latest.integration.name` (e.g., `"5.0"`) +2. From Prow job name: extract version pattern (e.g., `4.15` from `pull-ci-...-4.15-e2e-aws`) +3. If both fail: log a warning but still attempt the query with a reasonable default (latest GA release) + +When `USE_RELEASE_CONTEXT=true`, the release version is already extracted in the Release Repo Discovery step. Use it directly. + +Classification: +- Pass rate >= 95% and now failing -> **likely genuine regression** +- Pass rate < 95% with `open_bugs > 0` -> **known flaky test** +- Pass rate < 95% with `open_bugs == 0` -> **unstable test** + +### Prow Job History / Pass Sequence Analysis + +Classify pass sequence pattern (left = newest): +- `FFFFFFFFFF` -> Permafail (High priority) +- `FFFFSSSSSS` -> Recent regression (High) +- `SSSSSFFFFF` -> Resolved (Low) +- `SFSFSFSFSF` -> Flaky (Medium) + +### Failure Output Consistency + +Compare error messages across multiple failures: +- **Highly consistent** (>90%): single cascading root cause +- **Moderately consistent** (50-90%): primary issue with secondaries +- **Inconsistent** (<50%): multiple issues or environmental instability + +### Disruption / Cluster Health Correlation (e2e jobs only, unless `--fast`) + +Check for cluster-level disruption: `ci-cluster-network-liveness` failures, operator degradation, etcd issues. + +--- + +## Root Cause Tracing Protocol + +For each failure, trace the root cause to its origin by reasoning through the available context layers. Do not use a fixed lookup table. Instead, follow this diagnostic decision tree and **cite the evidence at each step** so the reasoning is verifiable. + +### Step 1: Does the error reference a specific file? + +If the error message or stack trace points to a source file: + +- Is that file in `PR_CHANGED_FILES`? + - **Yes** → The PR likely introduced this issue. Fetch the on-demand diff for this file and correlate the error line with the change. + - **No** → Is the file in the operator repo (from operator context)? + - **Yes** → The issue is in pre-existing code, not caused by this PR. + - **No** → Is the file from a CI step container or build image? Check the step registry ref if available. + +### Step 2: Is the error about a missing tool, command, or image? + +If the error contains "command not found", "executable not found", "image not found", or similar: + +- Check the ci-operator config: what `container.from` or step `from:` image is used for this job? +- Is the missing tool expected to be in that image? +- Trace to: the ci-operator config (which image is specified) and optionally the Dockerfile that builds it. + +### Step 3: Is the error about authentication, credentials, or secrets? + +If the error contains "authentication failed", "unauthorized", "password is incorrect", "token expired", or similar: + +- Check the ci-operator config: does this job declare `credentials` entries? +- If yes, identify the credential name and namespace. The credential itself is stored externally (typically Vault) and injected by Prow. +- The fix is NOT in the code -- it's in the secret store or the ci-operator credential declaration. + +### Step 4: Is the error transient or environmental? + +If the error matches infrastructure patterns (network timeout, quota exceeded, lease exhaustion, registry errors): + +- No code or config fix is needed. The issue is in the CI environment or cloud provider. +- Recommend `/retest`. + +### Step 5: Is the error about missing generated code? + +If the build error references `zz_generated.deepcopy.go`, `zz_generated.defaults.go`, or CRD YAML files: + +- Check `PR_CHANGED_FILES`: were `_types.go` files changed? +- Were the generated files also updated? +- If types were changed but generated files were not → incomplete code generation. PR author needs to run `make generate && make manifests`. + +### Step 6: None of the above matched + +If the failure doesn't fit any of the patterns above, report it with: +- All available evidence (log excerpt, JUnit entry, step info) +- The context layers that were checked and what they showed +- Confidence: low +- A recommendation for manual investigation + +### Output Format (MANDATORY) + +Every failure in the report MUST include a **Root Cause Trace**. Do NOT skip this section. Do NOT replace it with a free-form "Root Cause" paragraph. Use this exact structure: + +```text +Root Cause Trace: + 1. + 2. + 3. + ... + Fix location: + Fix owner: ) | credential owner | transient — /retest> + Confidence: +``` + +Rules: +- Every step in the trace MUST reference a concrete artifact (log line number, file path, config entry, PR change list result) +- Do NOT assert a location without citing what led to that conclusion +- If a failure involves credentials from the ci-operator config, explicitly state the credential name, namespace, and that it is declared in `openshift/release` (not in the PR's code) +- If a job is marked `optional: true` in the ci-operator config, do NOT label it as a "Blocker" -- label it as "Non-blocking (optional)" + +--- + +## Stage-Aware Summary Logic + +If exactly three PRs were provided, summarize by stage: +- **PR #1 (API)**: schema/codegen/validation risks, boilerplate/generation drift +- **PR #2 (Implementation)**: controller logic, build, unit test, RBAC consistency +- **PR #3 (E2E)**: scenario coverage, e2e environment, cluster install stability + +When PR change context is available, validate staging: +- PR #1 should primarily contain API type changes (`_types.go`, CRD YAML). Flag if it contains controller changes. +- PR #2 should primarily contain controller/reconciler changes. Flag if it modifies API types (should be in PR #1). +- PR #3 should primarily contain test files (`_test.go`, `e2e/`). Flag if it contains production code. + +Cross-stage dependency detection: +- PR #2 compile failure referencing PR #1 types -> fix PR #1 first +- PR #3 "CRD not found" -> PRs #1/#2 not merged yet +- PR #2 build error in a file from `PR_CHANGED_FILES` -> directly caused by this PR +- PR #2 build error in a file NOT in `PR_CHANGED_FILES` -> likely dependency on PR #1 or generated code drift + +--- + +## Report Template + +Return a structured markdown report. When release context is available, include enriched job metadata. + +```text +=== CI Monitor Report === + +Repository: +Go Module: | Framework: (when operator context available) +PR Head SHA: +Monitoring: adaptive polling (60s/120s/60s) | Timeout: m +Fix Round: of | SHA Changes Detected: +Signals Observed: checks, status contexts +Release Context: +OCP Release: (when release context available) +Mode: + +──────────────────────────────────────── +PR Results +──────────────────────────────────────── +PR # "" — PASS | FAIL | TIMED_OUT + Checks: <pass>/<total> passed + Prow: <pass>/<total> passed + Failed: <list of failed context names> + +──────────────────────────────────────── +Failure Analysis +──────────────────────────────────────── +1) [PR #<n>] <check-name> + Provider: <github-actions | prow-status-context> + Failure Mode: <install | test | build | lint/boilerplate | infra-flake> + Required: <yes | no (optional)> (when release context available) + Cluster: <aws | gcp | azure | none> (when release context available) + Step: <step-ref-name> (when release context available) + Commands: <actual command or script> (when release context available) + Job URL: <prow view url or actions url> + Artifacts: <gcsweb artifact browser url> + + Evidence: + <key log excerpt — max 20 lines> + + JUnit Summary (if available): + Total: <N> | Passed: <N> | Failed: <N> | Skipped: <N> + + Sippy Flake Check (if available): + - <test name>: pass_rate=<N>% trend=<dir> open_bugs=<N> + + Root Cause Trace: + 1. <evidence point — what the error says> + 2. <evidence point — where it originates (PR code? repo? CI config? Vault?)> + 3. <evidence point — what context layers confirm> + Fix location: <specific file, config, or system> + Fix owner: <who can make the change> + + Root Cause Hypothesis: <text> + Confidence: <high | medium | low> + Fixable by agent: <yes | no — reason> + Error Signature: <hash> (for fix loop tracking) + + Suggested Fixes: + 1. <most targeted fix> + 2. <alternative> + + Validation: + - <command to verify fix locally> + - <command to rerun CI> + +──────────────────────────────────────── +Prow Job Breakdown +──────────────────────────────────────── +| Context | State | Mode | Required | Flake? | Action | +|---|---|---|---|---|---| +| ci/prow/<name> | failure | test | yes | no (98%) | fix required | +| ci/prow/<name> | failure | infra | yes | yes (72%) | /retest | +| ci/prow/<name> | success | — | no (optional) | — | — | +| tide | pending | gate | — | — | needs: lgtm, approved | + +──────────────────────────────────────── +Recommended Next Actions +──────────────────────────────────────── +1. <highest priority fix> +2. <second action> +3. <rerun plan> + +──────────────────────────────────────── +Auto-Retest Log (when auto-retest triggered) +──────────────────────────────────────── +| # | PR | Timestamp | Contexts Retested | Outcome | +|---|---|---|---|---| +| 1 | #342 | 12:45 | e2e-aws, e2e-gcp | passed on retry | +| 2 | #342 | 13:10 | upgrade | still failing (infra) | + +──────────────────────────────────────── +API Budget +──────────────────────────────────────── +Total GitHub API calls: <N> +Release context calls: <N> +PR change context calls: <N> +Auto-retest comment calls: <N> +Polling rounds: <N> +``` + +### Post Report as PR Comment (`--post-comment`) + +When `--post-comment` is set, after generating the report above, post it as a GitHub PR comment for each monitored PR: + +```bash +# Only post if gh is authenticated and we have a PR number +if [ "$POST_COMMENT" = true ]; then + for PR_NUMBER in "${PR_NUMBERS[@]}"; do + gh pr comment "$PR_NUMBER" --repo "$REPO" --body "$REPORT" + echo "[ci-monitor] Posted report as comment on PR #$PR_NUMBER" + done +fi +``` + +The full report (from `=== CI Monitor Report ===` through `API Budget`) is posted as the comment body. GitHub renders the markdown natively. + +If the comment exceeds GitHub's 65536 character limit, truncate the Evidence sections (keeping the Root Cause Trace and Suggested Fix for each job) and append a note: `_Report truncated. See full output in CI job logs._` + +--- + +## Fix-Push-Rewatch Protocol + +When `--max-fix-rounds > 0` and the analysis identifies fixable failures. + +### Eligibility + +Only attempt auto-fix when ALL conditions are met: +- Failure Mode is B (test), C (build), or D (lint/boilerplate) +- Root cause hypothesis has `high` or `medium` confidence +- The fix can be applied to files in the current working directory +- When operator repo context available: the fix targets files within the detected framework pattern + +### Branch Verification + +Before applying any fix, verify the local checkout matches the PR branch: + +```bash +PR_BRANCH=$(gh pr view "$PR_NUMBER" --repo "$REPO" --json headRefName --jq '.headRefName') +CURRENT_BRANCH=$(git branch --show-current) + +if [ "$CURRENT_BRANCH" != "$PR_BRANCH" ]; then + echo "Switching to PR branch: $PR_BRANCH" + git fetch origin "$PR_BRANCH" + git checkout "$PR_BRANCH" + git pull origin "$PR_BRANCH" +fi +``` + +If the branch cannot be checked out (not a local clone, permission issues), skip the fix loop and produce a report-only output. + +### Apply Fix + +1. Make the code change based on the suggested fix. +2. Run local verification. When operator repo context available, use discovered capabilities: + - If `HAS_MAKEFILE=true` and Makefile has `verify` target: `make verify` + - Always: `go build ./...`, `go vet ./...` + - For Mode D (lint): `make lint` or `make verify` depending on what the job runs (from release context `commands` field) +3. If local verification fails, revert and report without pushing. + +### Commit and Push + +```bash +git add -A +git commit -m "fix: <description of CI fix>" +git push +``` + +### Error Signature Tracking + +To determine if a fix attempt resolved the issue, track error signatures across rounds: + +1. **Extract error signature** from each failed context: take the first meaningful error line from the build log or JUnit failure message. +2. **Normalize**: strip timestamps, line numbers, hex addresses (`0x[a-f0-9]+`), UUIDs (`[a-f0-9-]{36}`), temp file paths (`/tmp/[^ ]+`). +3. **Hash**: `echo "$NORMALIZED_ERROR" | sha256sum | cut -c1-16` +4. **Store**: maintain a map of `context_name -> error_hash` per fix round. + +### Same-Error Detection + +After each fix round completes and new failures are collected: + +```bash +SAME_COUNT=0 +TOTAL_FAILED=0 +for CONTEXT in $(failed_contexts); do + TOTAL_FAILED=$((TOTAL_FAILED + 1)) + PREV_HASH=$(get_prev_round_hash "$CONTEXT") + CURR_HASH=$(get_curr_round_hash "$CONTEXT") + if [ "$PREV_HASH" = "$CURR_HASH" ] && [ -n "$PREV_HASH" ]; then + SAME_COUNT=$((SAME_COUNT + 1)) + fi +done + +SAME_RATIO=$((SAME_COUNT * 100 / TOTAL_FAILED)) +if [ "$SAME_RATIO" -ge 75 ]; then + echo "Fix ineffective: $SAME_RATIO% of failures have identical error signatures." + echo "Stopping fix loop." + # Produce final report +fi +``` + +If >= 75% of failed contexts share the same error hash as the previous round, classify as "same error" and stop the fix loop. + +### Termination + +- `FIX_ATTEMPT >= max-fix-rounds`: stop, produce final report with all rounds summarized +- New round passes: report success +- New round fails with SAME error (>= 75% hash match): stop, fix didn't work +- New round fails with a DIFFERENT error: attempt another fix if rounds remain + +### Never Auto-Fix + +- Install failures (Mode A) -- require cluster-level investigation +- Infra flakes (Mode E) -- recommend `/retest` only +- Repo-wide failures (same error across all open PRs) -- not caused by this PR +- Low-confidence hypotheses -- report only, let user decide + +--- + +## Integration Notes + +This skill is invoked by the `/oape:ci-monitor` command and follows this flow: + +1. Command validates inputs and resolves PRs (Phase 0) +2. Command gathers operator repo context (Phase 0, Precheck 4) +3. Command gathers PR change context (Phase 0, Precheck 5) +4. Skill fetches release repo context (ci-operator config + step registry) +5. Skill runs adaptive polling with SHA/retest tracking +6. Skill collects failure evidence (GCS artifacts, JUnit, logs, on-demand diffs) +7. Skill classifies failure modes using config-aware rules +8. Skill performs deep analysis (Sippy, history, consistency) +9. Skill traces root cause for each failure through context layers (PR changes, repo, release config, infrastructure) +10. Skill produces stage-aware summary (when 3 PRs) +11. Skill generates structured report with root cause traces +12. Skill executes auto-retest (when all failures are infra flakes) +13. Skill executes fix-push-rewatch loop (when enabled and fixable failures exist) + +The skill receives from the command: +- Resolved `REPO`, PR numbers, and all parsed flags +- Operator repo context: `GO_MODULE`, `FRAMEWORK`, `TEST_DIRS`, `HAS_MAKEFILE`, `OPERATOR_CONTEXT_SOURCE` (`local` | `github` | `none`) + - When `local`: full repo access for fix loop (can run `make`, edit files, push) + - When `github`: read-only context (framework, module path, test dirs) for analysis and reporting; fix loop requires local clone + - When `none`: skill falls back to framework-agnostic analysis +- PR change context: `PR_CHANGED_FILES` (list of file paths changed by each PR) and change type counts (`api`, `controller`, `test`, `crd`, `rbac`) + - Used in failure classification to correlate errors with changed files + - Used in fix loop to target fixes at files the PR actually touches + - Used in stage-aware summary to validate PR staging (e.g., PR #1 should be API-only) + +--- + +*This skill is part of the OAPE AI E2E Feature Development toolkit.* diff --git a/scripts/ci-monitor/dispatch.sh b/scripts/ci-monitor/dispatch.sh new file mode 100755 index 0000000..05895f7 --- /dev/null +++ b/scripts/ci-monitor/dispatch.sh @@ -0,0 +1,289 @@ +#!/usr/bin/env bash +# dispatch.sh — Reads ci-monitor-result.json and invokes further oape-ai-e2e +# tools based on the failure classification. +# +# This script is the bridge between "CI monitoring" (monitor.sh) and +# "further processing" (auto-fix, Claude, retest). It runs immediately +# after monitor.sh in the same CI job, with the oape-ai-e2e repo cloned +# and available at OAPE_ROOT. +# +# Required environment: +# RESULT_FILE — Path to ci-monitor-result.json (default: /tmp/ci-monitor-result.json) +# +# Optional environment: +# OAPE_ROOT — Root of the cloned oape-ai-e2e repo (default: /app) +# DRY_RUN — If "true", log actions without executing them +# PHASE — Override dispatch phase (default: "2") +# RETEST_INFRA_FLAKES — If "true", post /test for infra flakes (default: "false") +# MAX_RETESTS_PER_RUN — Max retest comments per run (default: 2) +# WORK_DIR — Working directory with CI logs (default: /tmp/ci-monitor) + +set -euo pipefail + +RESULT_FILE="${RESULT_FILE:-/tmp/ci-monitor-result.json}" +OAPE_ROOT="${OAPE_ROOT:-/app}" +DRY_RUN="${DRY_RUN:-false}" +PHASE="${PHASE:-2}" +RETEST_INFRA_FLAKES="${RETEST_INFRA_FLAKES:-false}" +MAX_RETESTS_PER_RUN="${MAX_RETESTS_PER_RUN:-2}" +WORK_DIR="${WORK_DIR:-/tmp/ci-monitor}" +REPORT_MARKER="<!-- oape-ci-monitor -->" + +# --------------------------------------------------------------------------- +# Prechecks +# --------------------------------------------------------------------------- +if [[ ! -f "$RESULT_FILE" ]]; then + echo "[dispatch] No result file found at ${RESULT_FILE} — nothing to dispatch" + exit 0 +fi + +if ! command -v jq &>/dev/null; then + echo "[dispatch] ERROR: jq is not installed" >&2 + exit 1 +fi + +# --------------------------------------------------------------------------- +# Read result +# --------------------------------------------------------------------------- +OVERALL_STATUS=$(jq -r '.overall_status' "$RESULT_FILE") +TRIGGER_COUNT=$(jq '.trigger_actions | length' "$RESULT_FILE") +PR_URL=$(jq -r '.pr_url' "$RESULT_FILE") +OWNER=$(jq -r '.owner' "$RESULT_FILE") +REPO=$(jq -r '.repo' "$RESULT_FILE") +PR_NUMBER=$(jq -r '.pr_number' "$RESULT_FILE") + +echo "============================================" +echo " OAPE CI Monitor — Dispatch" +echo " PR: ${PR_URL}" +echo " Status: ${OVERALL_STATUS}" +echo " Trigger Actions: ${TRIGGER_COUNT}" +echo " Phase: ${PHASE}" +echo " Dry Run: ${DRY_RUN}" +echo " Retest Infra Flakes: ${RETEST_INFRA_FLAKES}" +echo "============================================" + +# --------------------------------------------------------------------------- +# If all passed, nothing to dispatch +# --------------------------------------------------------------------------- +if [[ "$OVERALL_STATUS" == "passed" ]]; then + echo "[dispatch] All CI checks passed — no further action needed" + exit 0 +fi + +if [[ "$TRIGGER_COUNT" -eq 0 ]]; then + echo "[dispatch] No trigger actions in result — nothing to dispatch" + exit 0 +fi + +# --------------------------------------------------------------------------- +# Pre-compute: are ALL failures infra-flakes? +# --------------------------------------------------------------------------- +ALL_INFRA_FLAKE="false" +non_retest_count=$(jq '[.trigger_actions[] | select(.action != "retest")] | length' "$RESULT_FILE") +if [[ "$non_retest_count" -eq 0 && "$TRIGGER_COUNT" -gt 0 ]]; then + ALL_INFRA_FLAKE="true" +fi + +# --------------------------------------------------------------------------- +# Dispatch each action +# --------------------------------------------------------------------------- +echo "[dispatch] Processing ${TRIGGER_COUNT} trigger action(s)..." +echo "" + +ACTIONS_TAKEN_FILE="${WORK_DIR}/dispatch-actions.txt" +: > "$ACTIONS_TAKEN_FILE" +RETEST_COUNT_FILE="${WORK_DIR}/retest-count.txt" +echo 0 > "$RETEST_COUNT_FILE" + +# Use process substitution instead of pipe to avoid subshell variable scoping +while IFS= read -r entry; do + action=$(echo "$entry" | jq -r '.action') + job=$(echo "$entry" | jq -r '.job') + + echo "[dispatch] Action: ${action} | Job: ${job}" + + case "$action" in + + # --- Retest: post /test for infra flakes --- + retest) + RETEST_COUNT=$(cat "$RETEST_COUNT_FILE") + if [[ "$RETEST_INFRA_FLAKES" != "true" ]]; then + echo " -> Auto-retest disabled (set RETEST_INFRA_FLAKES=true to enable)" + elif [[ "$ALL_INFRA_FLAKE" != "true" ]]; then + echo " -> Skipping retest: not all failures are infra-flakes (mixed failure types)" + elif [[ "$RETEST_COUNT" -ge "$MAX_RETESTS_PER_RUN" ]]; then + echo " -> Retest limit reached (${RETEST_COUNT}/${MAX_RETESTS_PER_RUN})" + else + # Extract short job name (strip pull-ci-<owner>-<repo>-<branch>- prefix) + # shellcheck disable=SC2001 + short_name=$(echo "$job" | sed "s/^pull-ci-${OWNER}-${REPO}-[^-]*-//") + if [[ -z "$short_name" || "$short_name" == "$job" ]]; then + echo " -> WARN: Could not extract short job name, falling back to /retest" + short_name="" + fi + + if [[ -n "$short_name" ]]; then + retest_cmd="/test ${short_name}" + else + retest_cmd="/retest" + fi + + if [[ "$DRY_RUN" != "true" ]]; then + echo " -> Posting '${retest_cmd}' for infra-flake: ${job}" + gh pr comment "$PR_NUMBER" --repo "${OWNER}/${REPO}" \ + --body "$retest_cmd" 2>/dev/null || true + echo "posted \`${retest_cmd}\` for infra-flake \`${job}\`" >> "$ACTIONS_TAKEN_FILE" + else + echo " -> DRY RUN: Would post '${retest_cmd}' for ${job}" + fi + echo $((RETEST_COUNT + 1)) > "$RETEST_COUNT_FILE" + fi + ;; + + # --- Auto-fix for lint failures --- + auto-fix-lint) + auto_fix_script="${OAPE_ROOT}/scripts/pr-agent/auto-fix.sh" + if [[ ! -x "$auto_fix_script" ]]; then + echo " -> Auto-fix script not found at ${auto_fix_script}" + else + echo " -> Running auto-fix for lint failure: ${job}" + fix_output="" + fix_args=(--pr-url "$PR_URL" --category lint-failure --job "$job" --log-dir "$WORK_DIR") + if [[ "$DRY_RUN" == "true" ]]; then + fix_args+=(--dry-run) + fi + + if fix_output=$("$auto_fix_script" "${fix_args[@]}" 2>&1); then + echo "$fix_output" + fix_sha=$(echo "$fix_output" | grep -oP 'Pushed fix: \K[a-f0-9]+' || true) + if [[ -n "$fix_sha" ]]; then + echo "auto-fixed \`lint-failure\` (commit ${fix_sha})" >> "$ACTIONS_TAKEN_FILE" + elif [[ "$DRY_RUN" != "true" ]]; then + echo "auto-fix attempted for \`lint-failure\` on \`${job}\` (no changes needed)" >> "$ACTIONS_TAKEN_FILE" + fi + else + echo "$fix_output" + echo " -> Auto-fix failed for ${job} (non-fatal, continuing)" + fi + fi + ;; + + # --- Auto-fix for generated files (make generate/manifests) --- + auto-fix-generated) + auto_fix_script="${OAPE_ROOT}/scripts/pr-agent/auto-fix.sh" + if [[ ! -x "$auto_fix_script" ]]; then + echo " -> Auto-fix script not found at ${auto_fix_script}" + else + echo " -> Running auto-fix for generated files: ${job}" + fix_output="" + fix_args=(--pr-url "$PR_URL" --category trivial-generated-files --job "$job" --log-dir "$WORK_DIR") + if [[ "$DRY_RUN" == "true" ]]; then + fix_args+=(--dry-run) + fi + + if fix_output=$("$auto_fix_script" "${fix_args[@]}" 2>&1); then + echo "$fix_output" + fix_sha=$(echo "$fix_output" | grep -oP 'Pushed fix: \K[a-f0-9]+' || true) + if [[ -n "$fix_sha" ]]; then + echo "auto-fixed \`trivial-generated-files\` (commit ${fix_sha})" >> "$ACTIONS_TAKEN_FILE" + elif [[ "$DRY_RUN" != "true" ]]; then + echo "auto-fix attempted for \`trivial-generated-files\` on \`${job}\` (no changes needed)" >> "$ACTIONS_TAKEN_FILE" + fi + else + echo "$fix_output" + echo " -> Auto-fix failed for ${job} (non-fatal, continuing)" + fi + fi + ;; + + # --- Investigate: Claude analysis for complex failures --- + investigate) + log_analyzer="${OAPE_ROOT}/scripts/pr-agent/log-analyzer.sh" + if [[ ! -x "$log_analyzer" ]]; then + echo " -> Log analyzer not found at ${log_analyzer}" + else + echo " -> Running Claude analysis for: ${job}" + analyze_output="" + analyze_args=(--pr-url "$PR_URL" --job "$job" --log-dir "$WORK_DIR") + if [[ -f "$RESULT_FILE" ]]; then + analyze_args+=(--result-file "$RESULT_FILE") + fi + if [[ "$DRY_RUN" == "true" ]]; then + analyze_args+=(--dry-run) + fi + + if analyze_output=$("$log_analyzer" "${analyze_args[@]}" 2>&1); then + echo "$analyze_output" + analysis_file="${WORK_DIR}/failure-analysis.json" + if [[ -f "$analysis_file" ]]; then + analysis_summary=$(jq -r '.analysis[] | select(.job != "") | "\(.mode) (\(.confidence)): \(.root_cause)"' "$analysis_file" 2>/dev/null | head -1 || true) + if [[ -n "$analysis_summary" && "$DRY_RUN" != "true" ]]; then + echo "analyzed \`${job}\`: ${analysis_summary}" >> "$ACTIONS_TAKEN_FILE" + fi + elif [[ "$DRY_RUN" != "true" ]]; then + echo "analyzed \`${job}\` — see Claude analysis in log" >> "$ACTIONS_TAKEN_FILE" + fi + else + echo "$analyze_output" + echo " -> Claude analysis failed for ${job} (non-fatal, continuing)" + fi + fi + ;; + + *) + echo " -> Unknown action: ${action} — skipping" + ;; + esac + + echo "" +done < <(jq -c '.trigger_actions[]' "$RESULT_FILE") + +# --------------------------------------------------------------------------- +# Post-dispatch report update +# --------------------------------------------------------------------------- +if [[ -s "$ACTIONS_TAKEN_FILE" && "$DRY_RUN" != "true" ]]; then + echo "[dispatch] Updating CI monitor report with actions taken..." + + existing_comment_id=$(gh api "repos/${OWNER}/${REPO}/issues/${PR_NUMBER}/comments" \ + --jq ".[] | select(.body | contains(\"${REPORT_MARKER}\")) | .id" 2>/dev/null | head -1 || true) + + if [[ -n "$existing_comment_id" ]]; then + existing_body=$(gh api "repos/${OWNER}/${REPO}/issues/comments/${existing_comment_id}" \ + --jq '.body' 2>/dev/null || true) + + actions_section=$'\n---\n### Actions Taken by oape-ci-monitor\n' + while IFS= read -r line; do + actions_section+="- ${line}"$'\n' + done < "$ACTIONS_TAKEN_FILE" + actions_section+=$'\n*Updated on '"$(date -u +'%Y-%m-%d %H:%M UTC')"'*' + + updated_body="${existing_body}${actions_section}" + gh api "repos/${OWNER}/${REPO}/issues/comments/${existing_comment_id}" \ + -X PATCH -f body="$updated_body" > /dev/null 2>&1 || true + echo "[dispatch] Report updated with actions taken" + else + echo "[dispatch] WARN: Could not find CI monitor comment to update" + fi +fi + +# --------------------------------------------------------------------------- +# Summary +# --------------------------------------------------------------------------- +echo "[dispatch] Dispatch complete" + +CATEGORY_SUMMARY=$(jq -r ' + .failure_categories + | to_entries + | map("\(.key): \(.value)") + | join(", ")' "$RESULT_FILE") + +echo "[dispatch] Failure categories: ${CATEGORY_SUMMARY}" + +if [[ -s "$ACTIONS_TAKEN_FILE" ]]; then + echo "[dispatch] Actions taken:" + while IFS= read -r line; do + echo " - ${line}" + done < "$ACTIONS_TAKEN_FILE" +else + echo "[dispatch] No actions were executed this run" +fi diff --git a/scripts/ci-monitor/monitor.sh b/scripts/ci-monitor/monitor.sh new file mode 100755 index 0000000..115fb93 --- /dev/null +++ b/scripts/ci-monitor/monitor.sh @@ -0,0 +1,1083 @@ +#!/usr/bin/env bash +# monitor.sh — OAPE CI Monitor. +# +# Analyzes CI failures on a PR: collects GCS artifacts for failed jobs, +# classifies failures deterministically, queries Sippy for flake history, +# and produces a structured failure analysis report. +# +# Supports two modes: +# Event-triggered (SKIP_POLL=true): fetches checks once, no polling. +# Polling (default): polls CI checks until all complete or timeout. +# +# Runs from any CI system (GitHub Actions, Prow, or locally). +# +# Required environment: +# PR_URL — Full GitHub PR URL (e.g. https://github.com/org/repo/pull/123) +# GH_TOKEN — GitHub token for API access +# +# Optional environment: +# SKIP_POLL — If "true", fetch checks once without polling (for event triggers) +# POLL_INTERVAL — Seconds between CI status polls (default: 120) +# POLL_TIMEOUT — Max seconds to wait for all checks (default: 7200) +# GCSWEB_BASE_URL — Base URL for gcsweb artifact access +# SIPPY_API_URL — Base URL for Sippy flake history API +# DRY_RUN — If "true", skip PR comment posting +# RESULT_FILE — Path for machine-readable JSON output +# SELF_JOB_NAME — This job's name in CI (excluded from monitoring) + +set -euo pipefail + +# shellcheck disable=SC2034 +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +# --------------------------------------------------------------------------- +# Configuration +# --------------------------------------------------------------------------- +PR_URL="${PR_URL:-}" +DRY_RUN="${DRY_RUN:-false}" +SKIP_POLL="${SKIP_POLL:-false}" +POLL_INTERVAL="${POLL_INTERVAL:-120}" +POLL_TIMEOUT="${POLL_TIMEOUT:-7200}" +GCSWEB_BASE_URL="${GCSWEB_BASE_URL:-https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com}" +SIPPY_API_URL="${SIPPY_API_URL:-https://sippy.dptools.openshift.org}" +SELF_JOB_NAME="${SELF_JOB_NAME:-oape-ci-monitor}" +RESULT_FILE="${RESULT_FILE:-/tmp/ci-monitor-result.json}" +WORK_DIR="${WORK_DIR:-/tmp/ci-monitor}" +REPORT_MARKER="<!-- oape-ci-monitor -->" + +# Release repo context (populated by fetch_release_context) +USE_RELEASE_CONTEXT="false" +RELEASE_VERSION="" + +# Parsed from PR_URL +OWNER="" +REPO="" +PR_NUMBER="" + +# --------------------------------------------------------------------------- +# Utility: retry with exponential backoff +# --------------------------------------------------------------------------- +gh_retry() { + local retries=3 delay=5 + for ((i = 1; i <= retries; i++)); do + if "$@"; then + return 0 + fi + if [[ "$i" -lt "$retries" ]]; then + echo "[retry] Attempt ${i}/${retries} failed, waiting ${delay}s..." >&2 + sleep "$delay" + delay=$((delay * 3)) + fi + done + echo "[retry] All ${retries} attempts failed for: $*" >&2 + return 1 +} + +# --------------------------------------------------------------------------- +# Parse PR URL into OWNER, REPO, PR_NUMBER +# --------------------------------------------------------------------------- +parse_pr_url() { + local url="$1" + if [[ "$url" =~ https://github.com/([^/]+)/([^/]+)/pull/([0-9]+) ]]; then + OWNER="${BASH_REMATCH[1]}" + REPO="${BASH_REMATCH[2]}" + PR_NUMBER="${BASH_REMATCH[3]}" + else + echo "ERROR: Invalid PR URL format: $url" >&2 + echo "Expected: https://github.com/{owner}/{repo}/pull/{number}" >&2 + exit 1 + fi +} + +# --------------------------------------------------------------------------- +# Prechecks +# --------------------------------------------------------------------------- +run_prechecks() { + echo "[precheck] Verifying prerequisites..." + + if [[ -z "$PR_URL" ]]; then + echo "ERROR: PR_URL environment variable is required" >&2 + exit 1 + fi + + if [[ -z "${GH_TOKEN:-}" ]]; then + echo "ERROR: GH_TOKEN environment variable is required" >&2 + exit 1 + fi + + if ! command -v gh &>/dev/null; then + echo "ERROR: gh CLI is not installed" >&2 + exit 1 + fi + + if ! command -v jq &>/dev/null; then + echo "ERROR: jq is not installed" >&2 + exit 1 + fi + + if ! gh auth status &>/dev/null; then + echo "ERROR: gh CLI is not authenticated" >&2 + exit 1 + fi + + mkdir -p "$WORK_DIR" + echo "[precheck] All prechecks passed" +} + +# =========================================================================== +# Phase 0: Fetch release repo context (ci-operator config from openshift/release) +# =========================================================================== +fetch_release_context() { + echo "[release-ctx] Fetching ci-operator config for ${OWNER}/${REPO}..." + + local base_branch + base_branch=$(gh pr view "$PR_NUMBER" --repo "${OWNER}/${REPO}" \ + --json baseRefName --jq '.baseRefName' 2>/dev/null || echo "") + + if [[ -z "$base_branch" ]]; then + echo "[release-ctx] Could not determine base branch, skipping release context" + return 0 + fi + + local config_base="https://raw.githubusercontent.com/openshift/release/master/ci-operator/config/${OWNER}/${REPO}" + local config_file="${OWNER}-${REPO}-${base_branch}.yaml" + local local_config="${WORK_DIR}/ci-operator-config.yaml" + + if ! curl -sf --max-time 15 "${config_base}/${config_file}" -o "$local_config" 2>/dev/null; then + config_file="${OWNER}-${REPO}-master.yaml" + if ! curl -sf --max-time 15 "${config_base}/${config_file}" -o "$local_config" 2>/dev/null; then + echo "[release-ctx] No ci-operator config found for ${OWNER}/${REPO}. Using name-based classification." + return 0 + fi + fi + + if [[ ! -s "$local_config" ]]; then + echo "[release-ctx] Config file empty, skipping" + return 0 + fi + + USE_RELEASE_CONTEXT="true" + echo "[release-ctx] Config fetched: ${config_file}" + + # Extract OCP release version for Sippy queries + # Try releases.latest.release.version, then releases.latest.integration.name + if command -v python3 &>/dev/null; then + RELEASE_VERSION=$(python3 -c " +import yaml, sys +try: + cfg = yaml.safe_load(open('$local_config')) + rels = cfg.get('releases', {}).get('latest', {}) + ver = rels.get('release', {}).get('version', '') + if not ver: + ver = rels.get('integration', {}).get('name', '') + print(ver) +except: + print('') +" 2>/dev/null || echo "") + fi + + if [[ -n "$RELEASE_VERSION" ]]; then + echo "[release-ctx] OCP release version: ${RELEASE_VERSION}" + fi + + # Build job manifest: tests[].as -> {optional, cluster_profile} + if command -v python3 &>/dev/null; then + python3 -c " +import yaml, json, sys +try: + cfg = yaml.safe_load(open('$local_config')) + manifest = {} + for test in cfg.get('tests', []): + name = test.get('as', '') + if not name: + continue + entry = { + 'optional': test.get('optional', False), + 'always_run': test.get('always_run', True), + 'cluster_profile': '', + } + steps = test.get('steps', {}) + if isinstance(steps, dict): + entry['cluster_profile'] = steps.get('cluster_profile', '') + manifest[name] = entry + json.dump(manifest, open('${WORK_DIR}/job-manifest.json', 'w'), indent=2) + print(f'[release-ctx] Job manifest: {len(manifest)} jobs parsed') +except Exception as e: + print(f'[release-ctx] Warning: could not parse job manifest: {e}', file=sys.stderr) + json.dump({}, open('${WORK_DIR}/job-manifest.json', 'w')) +" 2>/dev/null || echo '{}' > "${WORK_DIR}/job-manifest.json" + else + echo '{}' > "${WORK_DIR}/job-manifest.json" + echo "[release-ctx] python3 not available, skipping YAML parsing" + fi +} + +# =========================================================================== +# Phase 1: Poll CI checks until all complete (or timeout) +# =========================================================================== +fetch_ci_checks() { + local output_file="${WORK_DIR}/ci-checks.json" + + if ! gh_retry gh pr checks "$PR_NUMBER" --repo "${OWNER}/${REPO}" \ + --json name,state,link,bucket \ + > "$output_file" 2>/dev/null; then + echo "[]" > "$output_file" + fi + + # Filter out self and non-CI contexts (merge gates, review bots, etc.) + local filtered + filtered=$(jq --arg self "$SELF_JOB_NAME" \ + '[.[] | select( + .name != $self + and (.name | contains($self) | not) + and (.name | test("^(tide|Mergeable|DCO|CodeRabbit|stale|sonarcloud|codecov)"; "i") | not) + )]' \ + "$output_file") + echo "$filtered" > "$output_file" + + echo "$output_file" +} + +get_check_summary() { + local checks_file="$1" + local total passed failed pending + + total=$(jq 'length' "$checks_file") + passed=$(jq '[.[] | select(.bucket == "pass")] | length' "$checks_file") + failed=$(jq '[.[] | select(.bucket == "fail")] | length' "$checks_file") + pending=$(jq '[.[] | select(.bucket == "pending")] | length' "$checks_file") + + echo "${passed}/${total} passed | ${failed} failed | ${pending} pending" +} + +all_checks_complete() { + local checks_file="$1" + local pending + pending=$(jq '[.[] | select(.bucket == "pending")] | length' "$checks_file") + [[ "$pending" -eq 0 ]] +} + +poll_until_complete() { + echo "[poll] Waiting for all CI checks to complete (timeout: ${POLL_TIMEOUT}s, interval: ${POLL_INTERVAL}s)..." + + local elapsed=0 + local checks_file + + while true; do + checks_file=$(fetch_ci_checks) + local summary + summary=$(get_check_summary "$checks_file") + echo "[poll] [${elapsed}s] CI status: ${summary}" + + if all_checks_complete "$checks_file"; then + echo "[poll] All CI checks complete after ${elapsed}s" + return 0 + fi + + if [[ "$elapsed" -ge "$POLL_TIMEOUT" ]]; then + echo "[poll] Timeout reached (${POLL_TIMEOUT}s) — reporting on current state" + return 0 + fi + + sleep "$POLL_INTERVAL" + elapsed=$((elapsed + POLL_INTERVAL)) + done +} + +# =========================================================================== +# Phase 2: Collect GCS artifacts for failed jobs +# =========================================================================== +fetch_gcs_artifact() { + local job_name="$1" + local job_url="$2" + local log_file + log_file="${WORK_DIR}/log-$(echo "$job_name" | tr '/ ' '__').txt" + + if [[ "$job_url" == *"prow"* ]] || [[ "$job_url" == *"/view/gs/"* ]]; then + local gcs_path + gcs_path=$(echo "$job_url" | sed -n 's|.*/view/g[cs]s\?/||p') + if [[ -n "$gcs_path" ]]; then + local build_log_url="${GCSWEB_BASE_URL}/gcs/${gcs_path}/build-log.txt" + echo "[artifacts] Fetching build-log.txt for ${job_name}..." + if curl -sSL --max-time 30 "$build_log_url" 2>/dev/null | tail -2000 > "$log_file" 2>/dev/null; then + if [[ -s "$log_file" ]]; then + echo "[artifacts] Collected build-log.txt ($(wc -l < "$log_file") lines)" + + # Also try to fetch junit XML for test-level detail + local junit_url="${GCSWEB_BASE_URL}/gcs/${gcs_path}/artifacts/junit/" + curl -sSL --max-time 15 "$junit_url" 2>/dev/null \ + | grep -oP 'href="[^"]*\.xml"' \ + | head -5 \ + | sed 's/href="//;s/"//' \ + | while read -r xml_path; do + curl -sSL --max-time 15 "${junit_url}${xml_path}" 2>/dev/null \ + >> "${WORK_DIR}/junit-$(echo "$job_name" | tr '/ ' '__').xml" || true + done + return 0 + fi + fi + fi + fi + + if [[ "$job_url" == *"github.com"*"/actions/"* ]]; then + local run_id + run_id=$(echo "$job_url" | grep -oP 'runs/\K[0-9]+' || true) + if [[ -n "$run_id" ]]; then + echo "[artifacts] Fetching GHA failed step logs for ${job_name}..." + gh_retry gh run view "$run_id" --repo "${OWNER}/${REPO}" --log-failed \ + > "$log_file" 2>/dev/null || true + if [[ -s "$log_file" ]]; then + echo "[artifacts] Collected GHA logs ($(wc -l < "$log_file") lines)" + return 0 + fi + fi + fi + + echo "[artifacts] No logs collected for ${job_name}" + return 1 +} + +collect_failure_artifacts() { + local checks_file="${WORK_DIR}/ci-checks.json" + local collected=0 + + echo "[artifacts] Collecting artifacts for failed jobs..." + + jq -r '.[] | select(.bucket == "fail") | "\(.name)\t\(.link)"' "$checks_file" \ + | while IFS=$'\t' read -r job_name job_url; do + [[ -z "$job_name" ]] && continue + if fetch_gcs_artifact "$job_name" "${job_url:-}"; then + collected=$((collected + 1)) + fi + done + + echo "[artifacts] Collection complete" +} + +# =========================================================================== +# Phase 3: Deterministic failure classification +# =========================================================================== +classify_single_failure() { + local log_file="$1" + + if [[ ! -s "$log_file" ]]; then + echo "unknown" + return + fi + + local content + content=$(cat "$log_file") + + # Install failures (cluster provisioning) + if echo "$content" | grep -qiE \ + 'failed to install|cluster installation failed|install.*timed out|'\ + 'waiting for bootstrap|failed to create cluster|'\ + 'level=fatal.*installer|cluster creation failed|bootstrap.*timed out|'\ + 'waiting for bootstrapComplete'; then + echo "install-failure" + # Build / compile failures + elif echo "$content" | grep -qiE \ + 'cannot compile|undefined:|syntax error|cannot use.*as.*in|'\ + 'build.*failed|compilation error|cannot find package|imported and not used'; then + echo "build-failure" + # Generated files out of date (check before lint — generated-files errors + # often co-occur with lint markers but need a different fix command) + elif echo "$content" | grep -qiE \ + 'generated code is out of date|make generate|make manifests|deepcopy-gen|zz_generated'; then + echo "generated-files-failure" + # Lint / formatting / boilerplate failures + elif echo "$content" | grep -qiE \ + 'gofmt|goimports|formatting differs|golangci-lint|golint|staticcheck|revive|lint.*failed|boilerplate'; then + echo "lint-failure" + # Test failures + elif echo "$content" | grep -qiE \ + '--- FAIL|FAIL\s|panic:.*test|assertion failed|test.*failed'; then + echo "test-failure" + # Infrastructure / transient flakes + elif echo "$content" | grep -qiE \ + 'context deadline exceeded|connection refused|i/o timeout|ErrImagePull|ImagePullBackOff|'\ + 'pod sandbox|TLS handshake timeout|quota.*exceeded|unable to provision|'\ + 'registry\.ci\.openshift\.org.*(timeout|error)|'\ + 'etcdserver: request timed out|lease lost|'\ + 'error creating.*instance|InsufficientInstanceCapacity|'\ + 'unable to get lease|failed to acquire lease|'\ + 'dial tcp.*timeout'; then + echo "infra-flake" + else + echo "unknown" + fi +} + +classify_all_failures() { + local checks_file="${WORK_DIR}/ci-checks.json" + local analysis_file="${WORK_DIR}/failure-analysis.json" + local results="[]" + + echo "[classify] Classifying failures..." + + jq -r '.[] | select(.bucket == "fail") | .name' "$checks_file" \ + | while IFS= read -r job_name; do + [[ -z "$job_name" ]] && continue + local log_id + log_id=$(echo "$job_name" | tr '/ ' '__') + local log_file="${WORK_DIR}/log-${log_id}.txt" + + local category + category=$(classify_single_failure "$log_file") + + echo "${job_name} ${category}" + done > "${WORK_DIR}/classifications.tsv" + + # Build JSON from TSV + while IFS=$'\t' read -r job_name category; do + [[ -z "$job_name" ]] && continue + local log_id + log_id=$(echo "$job_name" | tr '/ ' '__') + local log_file="${WORK_DIR}/log-${log_id}.txt" + local job_url + job_url=$(jq -r --arg name "$job_name" '.[] | select(.name == $name) | .link' "$checks_file") + + local snippet="" + if [[ -s "$log_file" ]]; then + snippet=$(tail -20 "$log_file" | head -10 | tr '"' "'" | tr '\n' '|' | cut -c1-500) + fi + + results=$(echo "$results" | jq \ + --arg name "$job_name" \ + --arg cat "$category" \ + --arg url "$job_url" \ + --arg snip "$snippet" \ + '. + [{"job_name": $name, "category": $cat, "url": $url, "log_snippet": $snip, "flake_probability": 0}]') + done < "${WORK_DIR}/classifications.tsv" + + echo "$results" > "$analysis_file" + + local total_failures + total_failures=$(echo "$results" | jq 'length') + local by_category + by_category=$(echo "$results" | jq -r 'group_by(.category) | map("\(.[0].category): \(length)") | join(", ")') + echo "[classify] ${total_failures} failures classified: ${by_category}" +} + +# =========================================================================== +# Phase 4: Sippy flake history lookup +# =========================================================================== +resolve_release_version() { + local job_name="${1:-}" + + # 1. From ci-operator config (set by fetch_release_context) + if [[ -n "$RELEASE_VERSION" ]]; then + echo "$RELEASE_VERSION" + return + fi + + # 2. From Prow job name pattern (e.g., "4.18" from "pull-ci-...-4.18-e2e-aws") + if [[ -n "$job_name" ]]; then + local version + version=$(echo "$job_name" | grep -oP '\d+\.\d+' | head -1 || true) + if [[ -n "$version" ]]; then + echo "$version" + return + fi + fi + + echo "" +} + +query_sippy_flakes() { + local analysis_file="${WORK_DIR}/failure-analysis.json" + + if [[ ! -f "$analysis_file" ]]; then + return 0 + fi + + local test_failures + test_failures=$(jq -r '.[] | select(.category == "test-failure") | .job_name' "$analysis_file") + + if [[ -z "$test_failures" ]]; then + echo "[sippy] No test failures to check" + return 0 + fi + + local resolved_release + resolved_release=$(resolve_release_version "") + if [[ -n "$resolved_release" ]]; then + echo "[sippy] Using OCP release version: ${resolved_release}" + else + echo "[sippy] No release version resolved, queries may return incomplete data" + fi + + echo "[sippy] Querying Sippy for flake history..." + + while IFS= read -r job_name; do + [[ -z "$job_name" ]] && continue + + local release_for_job + release_for_job="${resolved_release:-$(resolve_release_version "$job_name")}" + local sippy_url + if [[ -n "$release_for_job" ]]; then + sippy_url="${SIPPY_API_URL}/api/tests?release=${release_for_job}&filter.test_name=${job_name}" + else + sippy_url="${SIPPY_API_URL}/api/jobs/flakes?job=${job_name}" + fi + local flake_data + flake_data=$(curl -sSL --max-time 10 "$sippy_url" 2>/dev/null || echo "{}") + + local flake_pct + flake_pct=$(echo "$flake_data" | jq -r '.flakePercentage // 0' 2>/dev/null || echo "0") + + if [[ "$flake_pct" != "0" && "$flake_pct" != "null" ]]; then + echo "[sippy] ${job_name}: ${flake_pct}% flake rate" + + # Update the analysis with flake probability + local updated + updated=$(jq --arg name "$job_name" --argjson pct "$flake_pct" \ + 'map(if .job_name == $name then .flake_probability = $pct else . end)' \ + "$analysis_file") + echo "$updated" > "$analysis_file" + + # Reclassify as infra-flake if flake rate is high (>30%) + if (( $(echo "$flake_pct > 30" | bc -l 2>/dev/null || echo 0) )); then + updated=$(jq --arg name "$job_name" \ + 'map(if .job_name == $name then .category = "infra-flake" else . end)' \ + "$analysis_file") + echo "$updated" > "$analysis_file" + echo "[sippy] ${job_name}: reclassified as infra-flake (flake rate ${flake_pct}%)" + fi + fi + done <<< "$test_failures" + + echo "[sippy] Flake history lookup complete" +} + +# =========================================================================== +# Phase 4b: Fetch PR change context (lazy — only when failures exist) +# =========================================================================== +fetch_pr_change_context() { + local context_file="${WORK_DIR}/pr-change-context.json" + + echo "[change-ctx] Fetching changed files for ${OWNER}/${REPO}#${PR_NUMBER}..." + + local changed_files + changed_files=$(gh pr view "$PR_NUMBER" --repo "${OWNER}/${REPO}" \ + --json files --jq '.files[].path' 2>/dev/null || true) + + if [[ -z "$changed_files" ]]; then + echo "[change-ctx] Could not fetch changed files" + echo '{"api":0,"controller":0,"test":0,"crd":0,"rbac":0,"other":0,"files":[]}' > "$context_file" + return 0 + fi + + local api=0 controller=0 test=0 crd=0 rbac=0 other=0 + local files_json="[]" + + while IFS= read -r f; do + [[ -z "$f" ]] && continue + files_json=$(echo "$files_json" | jq --arg f "$f" '. + [$f]') + case "$f" in + *_types.go|*types_*.go|*/api/*) api=$((api + 1)) ;; + *controller*|*reconcil*|*sync*.go) controller=$((controller + 1)) ;; + *_test.go|*_test.sh) test=$((test + 1)) ;; + *crd*.yaml|*crd*.json) crd=$((crd + 1)) ;; + *rbac*.yaml|*clusterrole*.yaml) rbac=$((rbac + 1)) ;; + *) other=$((other + 1)) ;; + esac + done <<< "$changed_files" + + jq -n \ + --argjson api "$api" \ + --argjson controller "$controller" \ + --argjson test "$test" \ + --argjson crd "$crd" \ + --argjson rbac "$rbac" \ + --argjson other "$other" \ + --argjson files "$files_json" \ + '{api:$api, controller:$controller, test:$test, crd:$crd, rbac:$rbac, other:$other, files:$files}' \ + > "$context_file" + + local total_files + total_files=$(echo "$files_json" | jq 'length') + echo "[change-ctx] PR changes: ${total_files} files (api:${api} controller:${controller} test:${test} crd:${crd} rbac:${rbac} other:${other})" +} + +# =========================================================================== +# Phase 5: Generate structured report +# =========================================================================== +generate_report() { + local checks_file="${WORK_DIR}/ci-checks.json" + local analysis_file="${WORK_DIR}/failure-analysis.json" + local report_file="${WORK_DIR}/report.md" + + local total passed failed pending + total=$(jq 'length' "$checks_file") + passed=$(jq '[.[] | select(.bucket == "pass")] | length' "$checks_file") + failed=$(jq '[.[] | select(.bucket == "fail")] | length' "$checks_file") + pending=$(jq '[.[] | select(.bucket == "pending")] | length' "$checks_file") + + local pr_title + pr_title=$(gh_retry gh pr view "$PR_NUMBER" --repo "${OWNER}/${REPO}" \ + --json title -q '.title' 2>/dev/null || echo "unknown") + + { + echo "${REPORT_MARKER}" + echo "## CI Monitor Report: ${OWNER}/${REPO}#${PR_NUMBER}" + echo "" + echo "**PR:** [${pr_title}](${PR_URL})" + echo "**Monitored at:** $(date -u +'%Y-%m-%d %H:%M UTC')" + if [[ "$USE_RELEASE_CONTEXT" == "true" ]]; then + echo "**Release Context:** available | OCP version: ${RELEASE_VERSION:-unknown}" + fi + + # PR change summary + local context_file="${WORK_DIR}/pr-change-context.json" + if [[ -f "$context_file" ]]; then + local ctx_total + ctx_total=$(jq '[.api,.controller,.test,.crd,.rbac,.other] | add' "$context_file" 2>/dev/null || echo 0) + if [[ "$ctx_total" -gt 0 ]]; then + local ctx_api ctx_ctrl ctx_test ctx_crd ctx_rbac ctx_other + ctx_api=$(jq '.api' "$context_file") + ctx_ctrl=$(jq '.controller' "$context_file") + ctx_test=$(jq '.test' "$context_file") + ctx_crd=$(jq '.crd' "$context_file") + ctx_rbac=$(jq '.rbac' "$context_file") + ctx_other=$(jq '.other' "$context_file") + local parts=() + [[ "$ctx_api" -gt 0 ]] && parts+=("${ctx_api} API") + [[ "$ctx_ctrl" -gt 0 ]] && parts+=("${ctx_ctrl} controller") + [[ "$ctx_test" -gt 0 ]] && parts+=("${ctx_test} test") + [[ "$ctx_crd" -gt 0 ]] && parts+=("${ctx_crd} CRD") + [[ "$ctx_rbac" -gt 0 ]] && parts+=("${ctx_rbac} RBAC") + [[ "$ctx_other" -gt 0 ]] && parts+=("${ctx_other} other") + local parts_str + parts_str=$(IFS=', '; echo "${parts[*]}") + echo "**PR changes:** ${ctx_total} files (${parts_str})" + fi + fi + echo "" + + # Overall status — check optional-only failures + local optional_failures=0 + local required_failures_count=0 + if [[ -f "$analysis_file" && "$failed" -gt 0 ]]; then + local job_manifest="${WORK_DIR}/job-manifest.json" + if [[ "$USE_RELEASE_CONTEXT" == "true" && -f "$job_manifest" ]]; then + jq -r '.[] | .job_name' "$analysis_file" 2>/dev/null | while IFS= read -r jn; do + local short + # shellcheck disable=SC2001 + short=$(echo "$jn" | sed "s/^pull-ci-${OWNER}-${REPO}-[^-]*-//") + local is_opt + is_opt=$(jq -r --arg n "$short" '.[$n].optional // false' "$job_manifest" 2>/dev/null || echo "false") + if [[ "$is_opt" == "true" ]]; then + optional_failures=$((optional_failures + 1)) + else + required_failures_count=$((required_failures_count + 1)) + fi + done + fi + fi + + if [[ "$failed" -eq 0 && "$pending" -eq 0 ]]; then + echo "**All ${total} CI checks passed.**" + echo "" + else + echo "### CI Check Summary" + echo "" + echo "| Status | Count |" + echo "|--------|-------|" + echo "| Passed | ${passed} |" + echo "| Failed | ${failed} |" + echo "| Pending | ${pending} |" + echo "| **Total** | **${total}** |" + echo "" + fi + + # Failed checks with classification + if [[ "$failed" -gt 0 && -f "$analysis_file" ]]; then + echo "### Failure Analysis" + echo "" + echo "| Job | Category | Flake% | Link |" + echo "|-----|----------|--------|------|" + + jq -r '.[] | "| \(.job_name) | `\(.category)` | \(.flake_probability)% | [logs](\(.url)) |"' \ + "$analysis_file" 2>/dev/null || true + echo "" + + # Infra flakes section + local flake_count + flake_count=$(jq '[.[] | select(.category == "infra-flake")] | length' "$analysis_file" 2>/dev/null || echo 0) + if [[ "$flake_count" -gt 0 ]]; then + echo "### Infrastructure Flakes (${flake_count})" + echo "" + echo "These failures appear to be infrastructure-related (timeouts, networking, quota) rather than code issues." + echo "Consider retesting with \`/retest\` or \`/test <job-name>\`." + echo "" + jq -r '.[] | select(.category == "infra-flake") | "- **\(.job_name)** — flake rate: \(.flake_probability)%"' \ + "$analysis_file" 2>/dev/null || true + echo "" + fi + + # Actionable failures + local actionable_count + actionable_count=$(jq '[.[] | select(.category != "infra-flake")] | length' "$analysis_file" 2>/dev/null || echo 0) + if [[ "$actionable_count" -gt 0 ]]; then + echo "### Actionable Failures (${actionable_count})" + echo "" + + # Group by category + for cat in "build-failure" "lint-failure" "generated-files-failure" "test-failure" "install-failure" "unknown"; do + local cat_items + cat_items=$(jq -r --arg c "$cat" '.[] | select(.category == $c) | .job_name' "$analysis_file" 2>/dev/null || true) + if [[ -n "$cat_items" ]]; then + echo "**${cat}:**" + while IFS= read -r name; do + echo "- ${name}" + done <<< "$cat_items" + echo "" + fi + done + fi + + # Log snippets for actionable failures + local has_snippets=false + while IFS= read -r entry; do + local snippet + snippet=$(echo "$entry" | jq -r '.log_snippet') + if [[ -n "$snippet" && "$snippet" != "null" ]]; then + has_snippets=true + break + fi + done < <(jq -c '.[] | select(.category != "infra-flake")' "$analysis_file" 2>/dev/null) + + if [[ "$has_snippets" == "true" ]]; then + echo "<details>" + echo "<summary>Log snippets for actionable failures</summary>" + echo "" + jq -c '.[] | select(.category != "infra-flake" and .log_snippet != "")' "$analysis_file" 2>/dev/null \ + | while IFS= read -r entry; do + local name snippet + name=$(echo "$entry" | jq -r '.job_name') + snippet=$(echo "$entry" | jq -r '.log_snippet' | tr '|' '\n') + echo "**${name}:**" + echo '```' + echo "$snippet" + echo '```' + echo "" + done + echo "</details>" + echo "" + fi + fi + + # Prow Job Breakdown table (all checks, not just failures) + echo "### Prow Job Breakdown" + echo "" + echo "| Job | State | Category | Required | Flake% | Action |" + echo "|-----|-------|----------|----------|--------|--------|" + + local job_manifest="${WORK_DIR}/job-manifest.json" + + jq -r '.[] | "\(.name)\t\(.bucket)\t\(.link)"' "$checks_file" 2>/dev/null \ + | while IFS=$'\t' read -r jb_name jb_bucket _jb_link; do + [[ -z "$jb_name" ]] && continue + local jb_category="--" jb_required="--" jb_flake="--" jb_action="--" + + # For failed jobs, look up category and flake% from analysis + if [[ "$jb_bucket" == "fail" && -f "$analysis_file" ]]; then + jb_category=$(jq -r --arg n "$jb_name" '.[] | select(.job_name == $n) | .category // "--"' "$analysis_file" 2>/dev/null || echo "--") + local fp + fp=$(jq -r --arg n "$jb_name" '.[] | select(.job_name == $n) | .flake_probability // 0' "$analysis_file" 2>/dev/null || echo "0") + if [[ "$fp" != "0" && "$fp" != "null" ]]; then + jb_flake="${fp}%" + fi + # Derive action from category + case "$jb_category" in + infra-flake) jb_action="/retest" ;; + lint-failure) jb_action="auto-fix-lint" ;; + generated-files-failure) jb_action="auto-fix-generated" ;; + build-failure|test-failure|install-failure) jb_action="investigate" ;; + unknown) jb_action="investigate" ;; + esac + fi + + # Look up required/optional from job manifest + if [[ "$USE_RELEASE_CONTEXT" == "true" && -f "$job_manifest" ]]; then + local short_name + # Pattern includes variable interpolation not supported by ${//} + # shellcheck disable=SC2001 + short_name=$(echo "$jb_name" | sed "s/^pull-ci-${OWNER}-${REPO}-[^-]*-//") + local is_optional + is_optional=$(jq -r --arg n "$short_name" '.[$n].optional // false' "$job_manifest" 2>/dev/null || echo "false") + if [[ "$is_optional" == "true" ]]; then + jb_required="no (optional)" + else + jb_required="yes" + fi + fi + + echo "| ${jb_name} | ${jb_bucket} | \`${jb_category}\` | ${jb_required} | ${jb_flake} | ${jb_action} |" + done + echo "" + + echo "---" + echo "*Generated by oape-ci-monitor on $(date -u +'%Y-%m-%d %H:%M UTC') | classification: deterministic (regex-based) | release context: ${USE_RELEASE_CONTEXT}*" + } > "$report_file" + + echo "[report] Report generated: ${report_file}" +} + +# =========================================================================== +# Phase 6: Post report as PR comment (idempotent update) +# =========================================================================== +post_report_comment() { + local report_file="${WORK_DIR}/report.md" + + if [[ ! -f "$report_file" ]]; then + echo "[post] No report file found" >&2 + return 1 + fi + + local body + body=$(cat "$report_file") + + if [[ "$DRY_RUN" == "true" ]]; then + echo "[post] DRY RUN: Would post/update report on ${OWNER}/${REPO}#${PR_NUMBER}" + echo "--- Report Preview ---" + cat "$report_file" + echo "--- End Preview ---" + return 0 + fi + + local existing_comment_id + existing_comment_id=$(gh api "repos/${OWNER}/${REPO}/issues/${PR_NUMBER}/comments" \ + --jq ".[] | select(.body | contains(\"${REPORT_MARKER}\")) | .id" 2>/dev/null | head -1 || true) + + if [[ -n "$existing_comment_id" ]]; then + gh_retry gh api "repos/${OWNER}/${REPO}/issues/comments/${existing_comment_id}" \ + -X PATCH -f body="$body" > /dev/null 2>&1 + echo "[post] Updated existing CI monitor comment on ${OWNER}/${REPO}#${PR_NUMBER}" + else + gh_retry gh pr comment "$PR_NUMBER" --repo "${OWNER}/${REPO}" --body "$body" > /dev/null 2>&1 + echo "[post] Posted new CI monitor comment on ${OWNER}/${REPO}#${PR_NUMBER}" + fi +} + +# =========================================================================== +# Phase 7: Write machine-readable result JSON ("trigger on failure" hook) +# =========================================================================== +write_result_json() { + local checks_file="${WORK_DIR}/ci-checks.json" + local analysis_file="${WORK_DIR}/failure-analysis.json" + local job_manifest="${WORK_DIR}/job-manifest.json" + + local total passed failed pending + total=$(jq 'length' "$checks_file") + passed=$(jq '[.[] | select(.bucket == "pass")] | length' "$checks_file") + failed=$(jq '[.[] | select(.bucket == "fail")] | length' "$checks_file") + pending=$(jq '[.[] | select(.bucket == "pending")] | length' "$checks_file") + + local failures="[]" + if [[ -f "$analysis_file" ]]; then + failures=$(cat "$analysis_file") + fi + + # Enrich each failure with optional metadata from the job manifest + if [[ "$USE_RELEASE_CONTEXT" == "true" && -f "$job_manifest" ]]; then + failures=$(echo "$failures" | jq --slurpfile manifest "$job_manifest" \ + --arg owner "$OWNER" --arg repo "$REPO" ' + map( + . as $f | + ($f.job_name | gsub("^pull-ci-" + $owner + "-" + $repo + "-[^-]+-"; "")) as $short | + ($manifest[0][$short].optional // false) as $opt | + . + {optional: $opt} + )') + else + failures=$(echo "$failures" | jq 'map(. + {optional: false})') + fi + + # Determine overall status — optional-only failures do not flip verdict + local overall_status="passed" + local required_failures + required_failures=$(echo "$failures" | jq '[.[] | select(.optional != true)] | length') + if [[ "$required_failures" -gt 0 ]]; then + overall_status="failed" + elif [[ "$failed" -gt 0 ]]; then + # All failures are optional + overall_status="passed-with-optional-failures" + fi + if [[ "$pending" -gt 0 && "$overall_status" != "failed" ]]; then + overall_status="pending" + fi + + # Compute category counts + local category_counts="{}" + if [[ "$failures" != "[]" ]]; then + category_counts=$(echo "$failures" | jq ' + group_by(.category) + | map({key: .[0].category, value: length}) + | from_entries') + fi + + # Include PR change context if available + local change_context="{}" + if [[ -f "${WORK_DIR}/pr-change-context.json" ]]; then + change_context=$(cat "${WORK_DIR}/pr-change-context.json") + fi + + jq -n \ + --arg pr_url "$PR_URL" \ + --arg owner "$OWNER" \ + --arg repo "$REPO" \ + --argjson pr_number "$PR_NUMBER" \ + --arg status "$overall_status" \ + --argjson total "$total" \ + --argjson passed "$passed" \ + --argjson failed "$failed" \ + --argjson pending "$pending" \ + --argjson failures "$failures" \ + --argjson categories "$category_counts" \ + --argjson change_context "$change_context" \ + --arg ts "$(date -u +'%Y-%m-%dT%H:%M:%SZ')" \ + '{ + pr_url: $pr_url, + owner: $owner, + repo: $repo, + pr_number: $pr_number, + overall_status: $status, + timestamp: $ts, + checks: { + total: $total, + passed: $passed, + failed: $failed, + pending: $pending + }, + failure_categories: $categories, + pr_change_context: $change_context, + failures: $failures, + trigger_actions: ( + if ($status == "failed" or $status == "passed-with-optional-failures") then + ($failures | map( + if .category == "infra-flake" then {action: "retest", job: .job_name, optional: .optional} + elif .category == "lint-failure" then {action: "auto-fix-lint", job: .job_name, optional: .optional} + elif .category == "generated-files-failure" then {action: "auto-fix-generated", job: .job_name, optional: .optional} + elif .category == "build-failure" then {action: "investigate", job: .job_name, optional: .optional} + elif .category == "test-failure" then {action: "investigate", job: .job_name, optional: .optional} + elif .category == "install-failure" then {action: "retest", job: .job_name, optional: .optional} + else {action: "investigate", job: .job_name, optional: .optional} + end + )) + else [] + end + ) + }' > "$RESULT_FILE" + + echo "[result] Machine-readable result written to ${RESULT_FILE}" + + # Log the trigger actions summary + local trigger_count + trigger_count=$(jq '.trigger_actions | length' "$RESULT_FILE") + if [[ "$trigger_count" -gt 0 ]]; then + echo "[result] Trigger actions suggested:" + jq -r '.trigger_actions[] | " - \(.action): \(.job) (optional: \(.optional))"' "$RESULT_FILE" + fi + + if [[ "$overall_status" == "passed-with-optional-failures" ]]; then + echo "[result] Note: all failures are optional — PR verdict is PASS" + fi +} + +# =========================================================================== +# Main +# =========================================================================== +main() { + echo "============================================" + echo " OAPE CI Monitor — Phase 2" + echo " PR: ${PR_URL}" + echo " Dry Run: ${DRY_RUN}" + echo " Skip Poll: ${SKIP_POLL}" + echo " Poll Interval: ${POLL_INTERVAL}s" + echo " Poll Timeout: ${POLL_TIMEOUT}s" + echo " Time: $(date -u +'%Y-%m-%d %H:%M UTC')" + echo "============================================" + + parse_pr_url "$PR_URL" + echo "[main] Monitoring ${OWNER}/${REPO}#${PR_NUMBER}" + + run_prechecks + + # Phase 0: Fetch release repo context + echo "" + echo "=== Phase 0: Fetch Release Repo Context ===" + fetch_release_context + + # Phase 1: Wait for CI checks to complete + echo "" + if [[ "$SKIP_POLL" == "true" ]]; then + echo "=== Phase 1: Fetch CI Checks (poll skipped — event-triggered) ===" + local checks_file_snap + checks_file_snap=$(fetch_ci_checks) + local snap_summary + snap_summary=$(get_check_summary "$checks_file_snap") + echo "[poll] CI status: ${snap_summary}" + else + echo "=== Phase 1: Poll CI Checks ===" + poll_until_complete + fi + + # Determine overall status + local checks_file="${WORK_DIR}/ci-checks.json" + local failed_count + failed_count=$(jq '[.[] | select(.bucket == "fail")] | length' "$checks_file") + + if [[ "$failed_count" -eq 0 ]]; then + echo "" + echo "=== All CI checks passed — generating summary report ===" + generate_report + post_report_comment + write_result_json + echo "" + echo "[main] CI monitor complete — all checks passed" + exit 0 + fi + + # Phase 2: Collect failure artifacts + echo "" + echo "=== Phase 2: Collect Failure Artifacts ===" + collect_failure_artifacts + + # Phase 3: Classify failures + echo "" + echo "=== Phase 3: Classify Failures ===" + classify_all_failures + + # Phase 4: Sippy flake lookup + echo "" + echo "=== Phase 4: Sippy Flake History ===" + query_sippy_flakes + + # Phase 4b: PR change context (lazy — only after classification) + echo "" + echo "=== Phase 4b: PR Change Context ===" + fetch_pr_change_context + + # Phase 5: Generate report + echo "" + echo "=== Phase 5: Generate Report ===" + generate_report + + # Phase 6: Post to PR + echo "" + echo "=== Phase 6: Post Report ===" + post_report_comment + + # Phase 7: Write machine-readable result + echo "" + echo "=== Phase 7: Write Result JSON ===" + write_result_json + + echo "" + echo "[main] CI monitor complete — ${failed_count} failure(s) analyzed" +} + +if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then + main +fi diff --git a/scripts/pr-agent/auto-fix.sh b/scripts/pr-agent/auto-fix.sh new file mode 100755 index 0000000..18b27d7 --- /dev/null +++ b/scripts/pr-agent/auto-fix.sh @@ -0,0 +1,343 @@ +#!/usr/bin/env bash +# auto-fix.sh — Standalone auto-fix engine for the OAPE CI Monitor. +# +# Clones a PR's repo, applies a deterministic fix for a specific failure +# category, verifies the fix compiles, and pushes it. Called by dispatch.sh +# after monitor.sh classifies failures. +# +# Usage: +# auto-fix.sh --pr-url <URL> --category <category> [options] +# +# Required: +# --pr-url <URL> PR URL (https://github.com/OWNER/REPO/pull/N) +# --category <cat> Fix category: trivial-format, trivial-import, +# trivial-lint, trivial-generated-files, +# lint-failure (coarse — refined via log analysis) +# +# Optional: +# --job <name> Prow job name (for audit logging) +# --log-dir <path> Directory containing CI log files (for fine-grained classification) +# --dry-run Show what would be done without committing/pushing + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +# shellcheck source=scripts/pr-agent/safety.sh +source "${SCRIPT_DIR}/safety.sh" + +# --------------------------------------------------------------------------- +# Configuration +# --------------------------------------------------------------------------- +BOT_USER="${BOT_USER:-openshift-app-platform-shift-bot}" +DRY_RUN="${DRY_RUN:-false}" +RUNNER_TEMP="${RUNNER_TEMP:-/tmp}" + +OWNER="" +REPO="" +PR_NUMBER="" +CURRENT_PR_URL="" + +# --------------------------------------------------------------------------- +# Usage +# --------------------------------------------------------------------------- +usage() { + echo "Usage: auto-fix.sh --pr-url <URL> --category <category> [--job <name>] [--log-dir <path>] [--dry-run]" + echo "" + echo "Categories: trivial-format, trivial-import, trivial-lint, trivial-generated-files, lint-failure" + exit 1 +} + +# --------------------------------------------------------------------------- +# Argument parsing +# --------------------------------------------------------------------------- +PR_URL_ARG="" +CATEGORY="" +JOB_NAME="" +LOG_DIR="" + +# shellcheck disable=SC2034 +while [[ $# -gt 0 ]]; do + case "$1" in + --pr-url) PR_URL_ARG="$2"; shift 2 ;; + --category) CATEGORY="$2"; shift 2 ;; + --job) JOB_NAME="$2"; shift 2 ;; + --log-dir) LOG_DIR="$2"; shift 2 ;; + --dry-run) DRY_RUN="true"; shift ;; + --help|-h) usage ;; + *) echo "[auto-fix] ERROR: Unknown argument: $1" >&2; usage ;; + esac +done + +if [[ -z "$PR_URL_ARG" ]]; then + echo "[auto-fix] ERROR: --pr-url is required" >&2 + usage +fi + +if [[ -z "$CATEGORY" ]]; then + echo "[auto-fix] ERROR: --category is required" >&2 + usage +fi + +case "$CATEGORY" in + trivial-format|trivial-import|trivial-lint|trivial-generated-files|lint-failure) ;; + *) + echo "[auto-fix] ERROR: Unsupported category: ${CATEGORY}" >&2 + echo "[auto-fix] Supported: trivial-format, trivial-import, trivial-lint, trivial-generated-files, lint-failure" >&2 + exit 1 + ;; +esac + +# --------------------------------------------------------------------------- +# parse_pr_url — extract owner/repo/pr_number from PR URL +# --------------------------------------------------------------------------- +parse_pr_url() { + local url="$1" + if [[ "$url" =~ https://github.com/([^/]+)/([^/]+)/pull/([0-9]+) ]]; then + OWNER="${BASH_REMATCH[1]}" + REPO="${BASH_REMATCH[2]}" + PR_NUMBER="${BASH_REMATCH[3]}" + elif [[ "$url" =~ ^([^/]+)/([^#]+)#([0-9]+)$ ]]; then + OWNER="${BASH_REMATCH[1]}" + REPO="${BASH_REMATCH[2]}" + PR_NUMBER="${BASH_REMATCH[3]}" + else + echo "[auto-fix] ERROR: Invalid PR URL format: $url" >&2 + exit 1 + fi + CURRENT_PR_URL="https://github.com/${OWNER}/${REPO}/pull/${PR_NUMBER}" +} + +# --------------------------------------------------------------------------- +# Main +# --------------------------------------------------------------------------- +parse_pr_url "$PR_URL_ARG" + +echo "============================================" +echo " OAPE Auto-Fix" +echo " PR: ${CURRENT_PR_URL}" +echo " Category: ${CATEGORY}" +echo " Job: ${JOB_NAME:-n/a}" +echo " Dry Run: ${DRY_RUN}" +echo "============================================" + +# --- Clone + checkout --- +workdir="${RUNNER_TEMP}/auto-fix-${OWNER}-${REPO}-${PR_NUMBER}" +if [[ -d "$workdir" ]]; then + rm -rf "$workdir" +fi + +echo "[auto-fix] Cloning ${OWNER}/${REPO}..." +if ! gh_retry gh repo clone "${OWNER}/${REPO}" "$workdir" -- --filter=blob:none --single-branch 2>/dev/null; then + echo "[auto-fix] ERROR: Failed to clone ${OWNER}/${REPO}" >&2 + audit_log "error" "$CATEGORY" "" "" "clone failed for ${OWNER}/${REPO}" + exit 1 +fi + +cd "$workdir" + +if ! gh pr checkout "$PR_NUMBER" 2>/dev/null; then + echo "[auto-fix] ERROR: Failed to checkout PR #${PR_NUMBER}" >&2 + audit_log "error" "$CATEGORY" "" "" "checkout failed for PR #${PR_NUMBER}" + exit 1 +fi + +git config user.name "$BOT_USER" +git config user.email "267347085+${BOT_USER}@users.noreply.github.com" +git remote set-url origin "https://x-access-token:${GH_TOKEN}@github.com/${OWNER}/${REPO}.git" + +base_branch=$(gh pr view "$PR_NUMBER" --repo "${OWNER}/${REPO}" --json baseRefName -q .baseRefName 2>/dev/null || echo "main") +git fetch origin "${base_branch}" --depth=1 2>/dev/null || true + +echo "[auto-fix] On branch: $(git branch --show-current), base: ${base_branch}" + +# --- Fine-grained classification from log files --- +# When dispatch.sh sends the coarse "lint-failure" category, refine it +# by reading the actual CI log files. +refine_lint_category() { + local log_dir="$1" + local job="$2" + + if [[ -z "$log_dir" || ! -d "$log_dir" ]]; then + echo "trivial-format" + return + fi + + local log_id + log_id=$(echo "$job" | tr '/ ' '__') + local log_file="${log_dir}/log-${log_id}.txt" + + if [[ ! -s "$log_file" ]]; then + for f in "${log_dir}"/log-*.txt; do + [[ -s "$f" ]] && log_file="$f" && break + done + fi + + if [[ ! -s "$log_file" ]]; then + echo "trivial-format" + return + fi + + local content + content=$(cat "$log_file") + + if echo "$content" | grep -qiE 'generated code is out of date|make generate|make manifests|deepcopy-gen|zz_generated|boilerplate'; then + echo "trivial-generated-files" + elif echo "$content" | grep -qiE 'imported and not used|could not import|import ordering'; then + echo "trivial-import" + elif echo "$content" | grep -qiE 'golangci-lint|golint|staticcheck|revive'; then + echo "trivial-lint" + elif echo "$content" | grep -qiE 'gofmt|goimports|formatting differs|diff.*\.go'; then + echo "trivial-format" + else + echo "trivial-format" + fi +} + +EFFECTIVE_CATEGORY="$CATEGORY" +if [[ "$CATEGORY" == "lint-failure" ]]; then + EFFECTIVE_CATEGORY=$(refine_lint_category "$LOG_DIR" "$JOB_NAME") + echo "[auto-fix] Refined lint-failure → ${EFFECTIVE_CATEGORY} (from log analysis)" +fi + +# --- Apply fix --- +echo "[auto-fix] Applying fix for: ${EFFECTIVE_CATEGORY}" + +changed_go_files=$(git diff --name-only "origin/${base_branch}" -- '*.go' 2>/dev/null || true) + +case "$EFFECTIVE_CATEGORY" in + trivial-format) + if [[ -n "$changed_go_files" ]]; then + echo "$changed_go_files" | xargs -r go fmt 2>/dev/null || true + if command -v goimports &>/dev/null; then + echo "$changed_go_files" | xargs -r goimports -w 2>/dev/null || true + fi + else + echo "[auto-fix] No Go files changed in this PR" + fi + ;; + + trivial-import) + if [[ -n "$changed_go_files" ]]; then + if command -v goimports &>/dev/null; then + echo "[auto-fix] Running goimports on PR-changed Go files" + echo "$changed_go_files" | xargs -r goimports -w 2>/dev/null || true + else + echo "[auto-fix] goimports not available, falling back to go fmt" + echo "$changed_go_files" | xargs -r go fmt 2>/dev/null || true + fi + else + echo "[auto-fix] No Go files changed in this PR" + fi + ;; + + trivial-lint) + if command -v golangci-lint &>/dev/null; then + echo "[auto-fix] Running golangci-lint --fix" + golangci-lint run --fix ./... 2>/dev/null || true + else + echo "[auto-fix] golangci-lint not available, skipping" + audit_log "skipped" "$EFFECTIVE_CATEGORY" "" "" "golangci-lint not available" + exit 0 + fi + ;; + + trivial-generated-files) + if [[ -f go.mod ]] && grep -q 'sigs.k8s.io/controller-runtime' go.mod; then + echo "[auto-fix] Detected controller-runtime, running make generate && make manifests" + make generate 2>/dev/null || true + make manifests 2>/dev/null || true + elif [[ -f go.mod ]] && grep -q 'github.com/openshift/library-go' go.mod; then + echo "[auto-fix] Detected library-go, running make update" + make update 2>/dev/null || true + else + echo "[auto-fix] Unknown framework, trying make generate then make update" + make generate 2>/dev/null || make update 2>/dev/null || true + fi + ;; +esac + +# --- Check for changes --- +modified_files=$(git diff --name-only; git ls-files --others --exclude-standard) +if [[ -z "$modified_files" ]]; then + echo "[auto-fix] No changes after applying ${EFFECTIVE_CATEGORY} fix" + audit_log "info" "$EFFECTIVE_CATEGORY" "" "" "no changes produced" + exit 0 +fi + +echo "[auto-fix] Modified files:" +echo "$modified_files" | while IFS= read -r f; do echo " $f"; done + +# --- Safety checks --- +if ! check_blocklist "$modified_files" "$EFFECTIVE_CATEGORY"; then + echo "[auto-fix] Blocklist violation on modified files, reverting" >&2 + git checkout -- . 2>/dev/null || true + git clean -fd 2>/dev/null || true + audit_log "reverted" "$EFFECTIVE_CATEGORY" "$modified_files" "" "post-fix blocklist violation" + exit 1 +fi + +if ! check_diff_size; then + echo "[auto-fix] Diff too large, reverting" >&2 + git checkout -- . 2>/dev/null || true + git clean -fd 2>/dev/null || true + audit_log "reverted" "$EFFECTIVE_CATEGORY" "$modified_files" "" "diff too large" + exit 1 +fi + +echo "[auto-fix] Verifying fix compiles..." +if ! go build ./... 2>/dev/null; then + echo "[auto-fix] Fix broke compilation, reverting" >&2 + git checkout -- . 2>/dev/null || true + git clean -fd 2>/dev/null || true + audit_log "reverted" "$EFFECTIVE_CATEGORY" "$modified_files" "" "fix broke compilation" + exit 1 +fi + +if ! go vet ./... 2>/dev/null; then + echo "[auto-fix] Fix failed go vet, reverting" >&2 + git checkout -- . 2>/dev/null || true + git clean -fd 2>/dev/null || true + audit_log "reverted" "$EFFECTIVE_CATEGORY" "$modified_files" "" "fix failed go vet" + exit 1 +fi + +# --- Dry-run gate --- +if [[ "$DRY_RUN" == "true" ]]; then + echo "[auto-fix] DRY RUN: Would commit and push fix for ${EFFECTIVE_CATEGORY}" + echo "[auto-fix] DRY RUN: Modified files:" + echo "$modified_files" | while IFS= read -r f; do echo " $f"; done + audit_log "dry-run" "$EFFECTIVE_CATEGORY" "$modified_files" "" "would commit and push" + git checkout -- . 2>/dev/null || true + git clean -fd 2>/dev/null || true + exit 0 +fi + +# --- Commit + push --- +if ! check_commit_limit 0; then + echo "[auto-fix] Commit limit reached, skipping push" >&2 + audit_log "skipped" "$EFFECTIVE_CATEGORY" "$modified_files" "" "commit limit reached" + git checkout -- . 2>/dev/null || true + git clean -fd 2>/dev/null || true + exit 1 +fi + +git diff --name-only -z | xargs -0 -r git add +git ls-files --others --exclude-standard -z | xargs -0 -r git add +git commit -m "fix: ${EFFECTIVE_CATEGORY} — auto-fix by oape-ci-monitor" + +sha=$(git rev-parse HEAD) + +if ! git pull --rebase origin HEAD 2>/dev/null; then + echo "[auto-fix] Rebase conflict — concurrent push detected, aborting" >&2 + git rebase --abort 2>/dev/null || true + git reset --hard HEAD~1 2>/dev/null || true + audit_log "reverted" "$EFFECTIVE_CATEGORY" "$modified_files" "$sha" "rebase conflict — concurrent push detected" + exit 1 +fi + +git push origin HEAD +increment_commit_count > /dev/null + +audit_log "auto-fix" "$EFFECTIVE_CATEGORY" "$modified_files" "$sha" "success" +echo "[auto-fix] Pushed fix: ${sha} (${EFFECTIVE_CATEGORY})" diff --git a/scripts/pr-agent/entrypoint.sh b/scripts/pr-agent/entrypoint.sh new file mode 100755 index 0000000..ee16f08 --- /dev/null +++ b/scripts/pr-agent/entrypoint.sh @@ -0,0 +1,986 @@ +#!/usr/bin/env bash +# entrypoint.sh — Main orchestration script for the OAPE PR Lifecycle Agent. +# +# Phase 1 MVP: periodic mode only, deterministic classification, +# auto-fix for trivial-format and trivial-generated-files. +# +# Usage: +# scripts/pr-agent/entrypoint.sh --mode periodic [--dry-run] +# scripts/pr-agent/entrypoint.sh --mode on-demand --pr-url <URL> [--dry-run] + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" + +# Source shared guardrail functions +# shellcheck source=scripts/pr-agent/safety.sh +source "${SCRIPT_DIR}/safety.sh" + +# --------------------------------------------------------------------------- +# Configuration +# --------------------------------------------------------------------------- +PR_AGENT_MAX_PRS="${PR_AGENT_MAX_PRS:-4}" +RATE_LIMIT_SECONDS="${RATE_LIMIT_SECONDS:-60}" +PR_TIMEOUT_SECONDS="${PR_TIMEOUT_SECONDS:-720}" +DRY_RUN="${DRY_RUN:-false}" +MONITOR_ONLY="${MONITOR_ONLY:-false}" +BOT_USER="${BOT_USER:-openshift-app-platform-shift-bot}" +GCSWEB_BASE_URL="${GCSWEB_BASE_URL:-https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com}" +TEAM_REPOS_CSV="${REPO_ROOT}/deploy/config/team-repos.csv" +RUNNER_TEMP="${RUNNER_TEMP:-/tmp}" + +# Per-PR context (set by parse_pr_url) +OWNER="" +REPO="" +PR_NUMBER="" +CURRENT_PR_URL="" + +# Track the PR list file location +PR_LIST_FILE="${RUNNER_TEMP}/pr-agent-pr-list.txt" + +# --------------------------------------------------------------------------- +# Crash handler — post error note to PR on unexpected failure +# --------------------------------------------------------------------------- +_cleanup_on_crash() { + local exit_code=$? + if [[ $exit_code -ne 0 && -n "$CURRENT_PR_URL" && "$DRY_RUN" != "true" ]]; then + local marker="<!-- oape-pr-agent-report -->" + local body + body="${marker} +## PR Agent Report + +**Status:** Agent encountered an error (exit code ${exit_code}). +Check the [CI run](${OAPE_RUN_URL:-https://github.com/${GITHUB_REPOSITORY:-unknown}/actions/runs/${GITHUB_RUN_ID:-${BUILD_ID:-0}}}) for details. + +--- +*Generated by oape-pr-agent on $(date -u +"%Y-%m-%d %H:%M UTC")*" + + gh pr comment "$PR_NUMBER" --repo "${OWNER}/${REPO}" --body "$body" 2>/dev/null || true + audit_log "error" "" "" "" "crash handler: exit code ${exit_code}" + fi +} +trap _cleanup_on_crash EXIT + +# =========================================================================== +# Argument Parsing +# =========================================================================== +MODE="" +PR_URL_ARG="" + +while [[ $# -gt 0 ]]; do + case "$1" in + --mode) + MODE="$2" + shift 2 + ;; + --pr-url) + PR_URL_ARG="$2" + shift 2 + ;; + --dry-run) + DRY_RUN="true" + shift + ;; + --monitor-only) + MONITOR_ONLY="true" + shift + ;; + *) + echo "ERROR: Unknown argument: $1" >&2 + exit 1 + ;; + esac +done + +if [[ -z "$MODE" ]]; then + echo "ERROR: --mode is required (periodic or on-demand)" >&2 + exit 1 +fi + +if [[ "$MODE" == "on-demand" && -z "$PR_URL_ARG" ]]; then + echo "ERROR: --pr-url is required for on-demand mode" >&2 + exit 1 +fi + +# =========================================================================== +# Prechecks +# =========================================================================== +run_prechecks() { + echo "[precheck] Verifying prerequisites..." + + if ! gh auth status &>/dev/null; then + echo "PRECHECK FAILED: gh CLI is not authenticated (run 'gh auth login')" >&2 + exit 1 + fi + + if [[ -z "${GH_TOKEN:-}" ]]; then + echo "PRECHECK FAILED: GH_TOKEN environment variable is not set" >&2 + exit 1 + fi + + if [[ ! -f "$TEAM_REPOS_CSV" ]]; then + echo "PRECHECK FAILED: team-repos.csv not found at ${TEAM_REPOS_CSV}" >&2 + exit 1 + fi + + echo "[precheck] All prechecks passed" +} + +# =========================================================================== +# PR URL Parsing +# =========================================================================== +parse_pr_url() { + local url="$1" + if [[ "$url" =~ https://github.com/([^/]+)/([^/]+)/pull/([0-9]+) ]]; then + OWNER="${BASH_REMATCH[1]}" + REPO="${BASH_REMATCH[2]}" + PR_NUMBER="${BASH_REMATCH[3]}" + elif [[ "$url" =~ ^([^/]+)/([^#]+)#([0-9]+)$ ]]; then + OWNER="${BASH_REMATCH[1]}" + REPO="${BASH_REMATCH[2]}" + PR_NUMBER="${BASH_REMATCH[3]}" + else + echo "PRECHECK FAILED: Invalid PR URL format: $url" >&2 + return 1 + fi + CURRENT_PR_URL="https://github.com/${OWNER}/${REPO}/pull/${PR_NUMBER}" +} + +# =========================================================================== +# Repo Allowlist Validation +# =========================================================================== +validate_repo_allowed() { + local owner="$1" repo="$2" + local target="https://github.com/${owner}/${repo}" + + if ! grep -q "${target}" "$TEAM_REPOS_CSV" 2>/dev/null; then + # Also check with .git suffix + if ! grep -q "${target}.git" "$TEAM_REPOS_CSV" 2>/dev/null; then + echo "PRECHECK FAILED: ${owner}/${repo} is not in the allowed repos list" >&2 + return 1 + fi + fi + return 0 +} + +# =========================================================================== +# PR Discovery (periodic mode) +# =========================================================================== +discover_oape_prs() { + echo "[discovery] Scanning repos for open PRs..." + true > "$PR_LIST_FILE" + + local repo_count=0 + local pr_count=0 + + { + read -r # Skip CSV header + while IFS=, read -r _product _role repo_url; do + local owner_repo + owner_repo=$(echo "$repo_url" | sed 's|https://github.com/||;s|\.git$||') + repo_count=$((repo_count + 1)) + + echo "[discovery] Checking ${owner_repo}..." + + local prs + if ! prs=$(gh_retry gh pr list --repo "$owner_repo" \ + --state open --json number,url,headRefName,title,labels --limit 20 2>/dev/null); then + echo "[discovery] WARNING: Failed to list PRs for ${owner_repo}, skipping" >&2 + continue + fi + + # Filter out PRs with pr-agent:skip label + prs=$(echo "$prs" | jq -r '[.[] | select(.labels | map(.name) | index("pr-agent:skip") | not)] | .[].url' 2>/dev/null || true) + + while IFS= read -r pr_url; do + [[ -z "$pr_url" ]] && continue + echo "$pr_url" >> "$PR_LIST_FILE" + pr_count=$((pr_count + 1)) + done <<< "$prs" + done + } < "$TEAM_REPOS_CSV" + + echo "[discovery] Found ${pr_count} open PR(s) across ${repo_count} repo(s)" +} + +# =========================================================================== +# Merge Conflict Detection +# =========================================================================== +check_merge_conflicts() { + local pr_url="$1" + local mergeable + mergeable=$(gh_retry gh pr view "$pr_url" --json mergeable -q .mergeable 2>/dev/null || echo "UNKNOWN") + if [[ "$mergeable" == "CONFLICTING" ]]; then + echo "[PR #${PR_NUMBER}] Merge conflict detected — skipping CI analysis and auto-fix" + return 1 + fi + return 0 +} + +# =========================================================================== +# CI Check Monitoring +# =========================================================================== +fetch_ci_status() { + local owner="$1" repo="$2" pr_number="$3" + local output_file="${RUNNER_TEMP}/ci-status-${owner}-${repo}-${pr_number}.json" + + # gh pr checks aggregates both GitHub Actions (Checks API) and Prow (Status API) + if ! gh_retry gh pr checks "$pr_number" --repo "${owner}/${repo}" \ + --json name,state,link,bucket \ + > "$output_file" 2>/dev/null; then + echo "[]" > "$output_file" + fi + + local total passed failed pending + total=$(jq 'length' "$output_file") + passed=$(jq '[.[] | select(.bucket == "pass")] | length' "$output_file") + failed=$(jq '[.[] | select(.bucket == "fail")] | length' "$output_file") + pending=$(jq '[.[] | select(.bucket == "pending")] | length' "$output_file") + + echo "[CI] ${passed}/${total} passed | ${failed} failed | ${pending} pending" +} + +aggregate_ci_status() { + local owner="$1" repo="$2" pr_number="$3" + local status_file="${RUNNER_TEMP}/ci-status-${owner}-${repo}-${pr_number}.json" + + local total passed failed pending + total=$(jq 'length' "$status_file") + passed=$(jq '[.[] | select(.bucket == "pass")] | length' "$status_file") + failed=$(jq '[.[] | select(.bucket == "fail")] | length' "$status_file") + pending=$(jq '[.[] | select(.bucket == "pending")] | length' "$status_file") + + if [[ "$total" -eq 0 ]]; then echo "no-checks" + elif [[ "$failed" -gt 0 ]]; then echo "some-failed" + elif [[ "$pending" -eq "$total" ]]; then echo "all-pending" + elif [[ "$pending" -gt 0 ]]; then echo "mixed-pending" + else echo "all-passed" + fi +} + +# =========================================================================== +# Failure Log Fetching +# =========================================================================== +fetch_failure_logs() { + local owner="$1" repo="$2" pr_number="$3" + local status_file="${RUNNER_TEMP}/ci-status-${owner}-${repo}-${pr_number}.json" + + jq -r '.[] | select(.bucket == "fail") | "\(.name)\t\(.link)"' "$status_file" \ + | while IFS=$'\t' read -r check_name url; do + [[ -z "$url" ]] && continue + + local log_id + log_id=$(echo "$check_name" | tr '/ ' '__') + + if [[ "$url" == *"github.com"*"/actions/"* ]]; then + # GitHub Actions: extract run ID, fetch failed step logs + local run_id + run_id=$(echo "$url" | grep -oP 'runs/\K[0-9]+' || true) + if [[ -n "$run_id" ]]; then + gh_retry gh run view "$run_id" --repo "${owner}/${repo}" --log-failed \ + > "${RUNNER_TEMP}/log-${log_id}.txt" 2>/dev/null || true + fi + elif [[ "$url" == *"prow"* ]]; then + # Prow: extract GCS path from the prow UI URL and fetch build-log.txt + local gcs_path + gcs_path=$(echo "$url" | sed -n 's|.*/view/g[cs]s\?/||p') + if [[ -n "$gcs_path" ]]; then + local gcsweb_url="${GCSWEB_BASE_URL}/gcs/${gcs_path}/build-log.txt" + curl -sSL "$gcsweb_url" 2>/dev/null | tail -1000 \ + > "${RUNNER_TEMP}/log-${log_id}.txt" 2>/dev/null || true + fi + fi + done +} + +# =========================================================================== +# Deterministic Failure Classification (regex-based, no Claude API) +# =========================================================================== +classify_failure_deterministic() { + local log_file="$1" + + if [[ ! -s "$log_file" ]]; then + echo "unknown" + return + fi + + local content + content=$(cat "$log_file") + + if echo "$content" | grep -qiE -- 'gofmt|goimports|formatting differs|diff.*\.go'; then + echo "trivial-format" + elif echo "$content" | grep -qiE -- 'imported and not used|could not import|import ordering'; then + echo "trivial-import" + elif echo "$content" | grep -qiE -- 'golangci-lint|golint|staticcheck|revive'; then + echo "trivial-lint" + elif echo "$content" | grep -qiE -- 'generated code is out of date|make generate|make manifests|deepcopy-gen|zz_generated'; then + echo "trivial-generated-files" + elif echo "$content" | grep -qiE -- 'cannot compile|undefined:|syntax error|cannot use.*as.*in'; then + echo "build-error" + elif echo "$content" | grep -qiE -- '--- FAIL|FAIL\s|panic:.*test|assertion failed'; then + echo "test-failure" + elif echo "$content" | grep -qiE -- 'context deadline exceeded|connection refused|i/o timeout|ErrImagePull|pod sandbox|TLS handshake timeout'; then + echo "infra-flake" + else + echo "unknown" + fi +} + +classify_all_failures() { + local owner="$1" repo="$2" pr_number="$3" + local analysis_file="${RUNNER_TEMP}/failure-analysis-${owner}-${repo}-${pr_number}.json" + local results="[]" + + for log_file in "${RUNNER_TEMP}"/log-*.txt; do + [[ -f "$log_file" ]] || continue + + local log_name + log_name=$(basename "$log_file" .txt | sed 's/^log-//') + + local category + category=$(classify_failure_deterministic "$log_file") + + results=$(echo "$results" | jq \ + --arg cat "$category" \ + --arg name "$log_name" \ + --arg file "$log_file" \ + '. + [{"category": $cat, "confidence": "high", "check_name": $name, "log_file": $file, "root_cause": $cat, "suggested_fix": ""}]') + done + + echo "$results" > "$analysis_file" + local trivial_count + trivial_count=$(echo "$results" | jq '[.[] | select(.category | startswith("trivial-"))] | length') + local total_count + total_count=$(echo "$results" | jq 'length') + echo "[classify] ${trivial_count}/${total_count} failures are trivially fixable" +} + +# =========================================================================== +# Auto-Fix Engine (Phase 1: trivial-format + trivial-generated-files only) +# =========================================================================== +apply_trivial_fixes() { + local owner="$1" repo="$2" pr_number="$3" + local analysis_file="${RUNNER_TEMP}/failure-analysis-${owner}-${repo}-${pr_number}.json" + + if [[ ! -f "$analysis_file" ]]; then + echo "[auto-fix] No failure analysis found, skipping" + return 0 + fi + + local fixable + fixable=$(jq -c '[.[] | select(.category == "trivial-format" or .category == "trivial-generated-files")]' "$analysis_file") + local fix_count + fix_count=$(echo "$fixable" | jq 'length') + + if [[ "$fix_count" -eq 0 ]]; then + echo "[auto-fix] No trivially fixable failures found" + return 0 + fi + + echo "[auto-fix] Found ${fix_count} fixable failure(s), cloning repo..." + + # Clone with blobless filter for performance + local workdir="${RUNNER_TEMP}/repo-${owner}-${repo}-${pr_number}" + if [[ -d "$workdir" ]]; then + rm -rf "$workdir" + fi + + if ! gh_retry gh repo clone "${owner}/${repo}" "$workdir" -- --filter=blob:none --single-branch 2>/dev/null; then + echo "[auto-fix] ERROR: Failed to clone ${owner}/${repo}" >&2 + audit_log "error" "" "" "" "clone failed for ${owner}/${repo}" + return 1 + fi + + cd "$workdir" + + if ! gh pr checkout "$pr_number" 2>/dev/null; then + echo "[auto-fix] ERROR: Failed to checkout PR #${pr_number}" >&2 + audit_log "error" "" "" "" "checkout failed for PR #${pr_number}" + cd "$REPO_ROOT" + return 1 + fi + + # Configure git identity for the bot + git config user.name "$BOT_USER" + git config user.email "267347085+${BOT_USER}@users.noreply.github.com" + git remote set-url origin "https://x-access-token:${GH_TOKEN}@github.com/${owner}/${repo}.git" + + # Determine PR base branch for scoping fixes + local base_branch + base_branch=$(gh pr view "$pr_number" --repo "${owner}/${repo}" --json baseRefName -q .baseRefName 2>/dev/null || echo "main") + git fetch origin "${base_branch}" --depth=1 2>/dev/null || true + + local pr_commit_count=0 + + echo "$fixable" | jq -c '.[]' | while IFS= read -r fix; do + local category + category=$(echo "$fix" | jq -r '.category') + + # Check global commit limits + if ! check_commit_limit "$pr_commit_count"; then + audit_log "skipped" "$category" "" "" "commit limit reached" + continue + fi + + echo "[auto-fix] Applying fix for: ${category}" + + case "$category" in + trivial-format) + # Scope go fmt to only PR-changed Go files + local changed_go_files + changed_go_files=$(git diff --name-only "origin/${base_branch}" -- '*.go' 2>/dev/null || true) + if [[ -n "$changed_go_files" ]]; then + echo "$changed_go_files" | xargs -r go fmt 2>/dev/null || true + if command -v goimports &>/dev/null; then + echo "$changed_go_files" | xargs -r goimports -w 2>/dev/null || true + fi + fi + ;; + trivial-generated-files) + # Framework-aware generation + if [[ -f go.mod ]] && grep -q 'sigs.k8s.io/controller-runtime' go.mod; then + make generate 2>/dev/null || true + make manifests 2>/dev/null || true + elif [[ -f go.mod ]] && grep -q 'github.com/openshift/library-go' go.mod; then + make update 2>/dev/null || true + else + make generate 2>/dev/null || make update 2>/dev/null || true + fi + ;; + *) + echo "[auto-fix] Skipping unsupported category: ${category}" + continue + ;; + esac + + # Check if there are actual changes + local modified_files + modified_files=$(git diff --name-only; git ls-files --others --exclude-standard) + if [[ -z "$modified_files" ]]; then + echo "[auto-fix] No changes after applying ${category} fix" + continue + fi + + # Pre-commit blocklist check on actual modified files + if ! check_blocklist "$modified_files" "$category"; then + echo "[auto-fix] Blocklist violation on modified files, reverting" + git checkout -- . 2>/dev/null || true + git clean -fd 2>/dev/null || true + audit_log "reverted" "$category" "$modified_files" "" "post-fix blocklist violation" + continue + fi + + # Diff size guard + if ! check_diff_size; then + git checkout -- . 2>/dev/null || true + git clean -fd 2>/dev/null || true + audit_log "reverted" "$category" "$modified_files" "" "diff too large" + continue + fi + + # Verify fix compiles + if ! go build ./... 2>/dev/null; then + echo "[auto-fix] Fix broke compilation, reverting" + git checkout -- . 2>/dev/null || true + git clean -fd 2>/dev/null || true + audit_log "reverted" "$category" "$modified_files" "" "fix broke compilation" + continue + fi + + if ! go vet ./... 2>/dev/null; then + echo "[auto-fix] Fix failed go vet, reverting" + git checkout -- . 2>/dev/null || true + git clean -fd 2>/dev/null || true + audit_log "reverted" "$category" "$modified_files" "" "fix failed go vet" + continue + fi + + if [[ "$DRY_RUN" == "true" ]]; then + echo "[auto-fix] DRY RUN: Would commit and push fix for ${category}" + audit_log "dry-run" "$category" "$modified_files" "" "would commit and push" + git checkout -- . 2>/dev/null || true + git clean -fd 2>/dev/null || true + continue + fi + + # Stage all changes (tracked + untracked) + git diff --name-only -z | xargs -0 -r git add + git ls-files --others --exclude-standard -z | xargs -0 -r git add + git commit -m "fix: ${category} — auto-fix by oape-pr-agent" + local sha + sha=$(git rev-parse HEAD) + + # Pull before push to handle concurrent pushes + if ! git pull --rebase origin HEAD 2>/dev/null; then + git rebase --abort 2>/dev/null || true + audit_log "reverted" "$category" "$modified_files" "$sha" "rebase conflict — concurrent push detected" + git reset --hard HEAD~1 2>/dev/null || true + continue + fi + + git push origin HEAD + pr_commit_count=$((pr_commit_count + 1)) + increment_commit_count > /dev/null + + audit_log "auto-fix" "$category" "$modified_files" "$sha" "success" + echo "[auto-fix] Pushed fix: ${sha} (${category})" + done + + cd "$REPO_ROOT" +} + +# =========================================================================== +# State Persistence (embedded in PR comment) +# =========================================================================== +load_persisted_state() { + local owner="$1" repo="$2" pr_number="$3" + local state_file="${RUNNER_TEMP}/pr-agent-state-${owner}-${repo}-${pr_number}.json" + + # Default empty state + if [[ ! -f "$state_file" ]]; then + echo '{"analyzed":[],"addressed":[],"last_run":""}' > "$state_file" + fi + + # Try to load state from existing PR comment + local marker="<!-- oape-pr-agent-report -->" + local existing_body + existing_body=$(gh api "repos/${owner}/${repo}/issues/${pr_number}/comments" \ + --jq ".[] | select(.body | contains(\"${marker}\")) | .body" 2>/dev/null | head -1 || true) + + if [[ -n "$existing_body" ]]; then + local persisted_state + persisted_state=$(echo "$existing_body" | grep -oP '(?<=oape-pr-agent-state:)[A-Za-z0-9+/=]+' | head -1 || true) + if [[ -n "$persisted_state" ]]; then + echo "$persisted_state" | base64 -d > "$state_file" 2>/dev/null || true + fi + fi +} + +save_persisted_state() { + local owner="$1" repo="$2" pr_number="$3" + local state_file="${RUNNER_TEMP}/pr-agent-state-${owner}-${repo}-${pr_number}.json" + + # Update last_run timestamp + local current_state + current_state=$(cat "$state_file" 2>/dev/null || echo '{"analyzed":[],"addressed":[],"last_run":""}') + echo "$current_state" | jq --arg ts "$(date -u +"%Y-%m-%dT%H:%M:%SZ")" '.last_run = $ts' > "$state_file" +} + +# =========================================================================== +# Status Report Generation +# =========================================================================== +generate_status_report() { + local owner="$1" repo="$2" pr_number="$3" + local report_file="${RUNNER_TEMP}/pr-agent-report-${owner}-${repo}-${pr_number}.md" + local ci_file="${RUNNER_TEMP}/ci-status-${owner}-${repo}-${pr_number}.json" + local analysis_file="${RUNNER_TEMP}/failure-analysis-${owner}-${repo}-${pr_number}.json" + local merge_conflict="${4:-false}" + + # Fetch PR metadata + local pr_info + pr_info=$(gh_retry gh pr view "$pr_number" --repo "${owner}/${repo}" \ + --json title,url,headRefName,baseRefName -q '.' 2>/dev/null || echo '{}') + + local title head base pr_url_display + title=$(echo "$pr_info" | jq -r '.title // "unknown"') + head=$(echo "$pr_info" | jq -r '.headRefName // "unknown"') + base=$(echo "$pr_info" | jq -r '.baseRefName // "unknown"') + pr_url_display="https://github.com/${owner}/${repo}/pull/${pr_number}" + + local run_link="${OAPE_RUN_URL:-https://github.com/${GITHUB_REPOSITORY:-unknown}/actions/runs/${GITHUB_RUN_ID:-${BUILD_ID:-0}}}" + local mode_label="${MODE:-periodic}" + local dry_run_banner="" + if [[ "$DRY_RUN" == "true" ]]; then + dry_run_banner="**DRY RUN** — no changes were made + +" + fi + + # CI stats + local passed=0 failed=0 pending=0 total=0 + if [[ -f "$ci_file" ]]; then + total=$(jq 'length' "$ci_file" 2>/dev/null || echo 0) + passed=$(jq '[.[] | select(.bucket == "pass")] | length' "$ci_file" 2>/dev/null || echo 0) + failed=$(jq '[.[] | select(.bucket == "fail")] | length' "$ci_file" 2>/dev/null || echo 0) + pending=$(jq '[.[] | select(.bucket == "pending")] | length' "$ci_file" 2>/dev/null || echo 0) + fi + + # Build report + { + echo "## PR Agent Report: ${REPO}#${pr_number}" + echo "" + echo "**PR:** [${title}](${pr_url_display})" + echo "**Branch:** \`${head}\` → \`${base}\`" + echo "**Run:** [#${GITHUB_RUN_ID:-${BUILD_ID:-N/A}}](${run_link})" + echo "**Mode:** ${mode_label}" + echo "" + echo "${dry_run_banner}" + + # Merge conflict section + if [[ "$merge_conflict" == "true" ]]; then + echo "### Merge Conflict" + echo "" + echo "> **This PR has merge conflicts.** Please rebase or merge the base branch to resolve conflicts before the agent can analyze CI failures or apply fixes." + echo "" + fi + + # CI check results + echo "### CI Check Results" + echo "" + echo "| Status | Count |" + echo "|--------|-------|" + echo "| Passed | ${passed} |" + echo "| Failed | ${failed} |" + echo "| Pending | ${pending} |" + echo "" + + if [[ "$merge_conflict" != "true" && -f "$ci_file" && "$failed" -gt 0 ]]; then + echo "<details>" + echo "<summary>Failed checks (${failed})</summary>" + echo "" + echo "| Check | Link |" + echo "|-------|------|" + jq -r '.[] | select(.bucket == "fail") | "| \(.name) | [\(.name)](\(.link)) |"' "$ci_file" 2>/dev/null || true + echo "" + echo "</details>" + echo "" + fi + + # Fixes applied + echo "### Fixes Applied" + echo "" + local fix_entries + fix_entries=$(grep '"action":"auto-fix"' "$AUDIT_LOG" 2>/dev/null || true) + if [[ -n "$fix_entries" ]]; then + echo "$fix_entries" | while IFS= read -r entry; do + local commit category outcome + commit=$(echo "$entry" | jq -r '.commit') + category=$(echo "$entry" | jq -r '.type') + outcome=$(echo "$entry" | jq -r '.outcome') + echo "- [\`${commit:0:7}\`](https://github.com/${owner}/${repo}/commit/${commit}) — \`${category}\`: ${outcome}" + done + else + echo "- (none)" + fi + echo "" + + # Infrastructure flakes + if [[ -f "$analysis_file" ]]; then + local flake_count + flake_count=$(jq '[.[] | select(.category == "infra-flake")] | length' "$analysis_file" 2>/dev/null || echo 0) + if [[ "$flake_count" -gt 0 ]]; then + echo "### Infrastructure Flakes" + echo "" + echo "The following CI jobs appear to be infrastructure flakes (timeouts, network errors) rather than code issues:" + echo "" + jq -r '.[] | select(.category == "infra-flake") | "- \(.check_name)"' "$analysis_file" 2>/dev/null || true + echo "" + fi + fi + + # Remaining issues + echo "### Remaining Issues" + echo "" + local blocked_entries + blocked_entries=$(grep -E '"action":"(blocked|skipped|reverted)"' "$AUDIT_LOG" 2>/dev/null || true) + if [[ -n "$blocked_entries" ]]; then + echo "$blocked_entries" | while IFS= read -r entry; do + local category outcome + category=$(echo "$entry" | jq -r '.type') + outcome=$(echo "$entry" | jq -r '.outcome') + echo "- \`${category}\`: ${outcome}" + done + fi + + # Non-trivial failures + if [[ -f "$analysis_file" ]]; then + local non_trivial + non_trivial=$(jq -r '.[] | select(.category == "build-error" or .category == "test-failure" or .category == "unknown") | "- `\(.category)`: \(.check_name)"' "$analysis_file" 2>/dev/null || true) + if [[ -n "$non_trivial" ]]; then + echo "" + echo "**Requires human attention:**" + echo "$non_trivial" + fi + fi + + if [[ -z "$blocked_entries" ]] && { [[ ! -f "$analysis_file" ]] || [[ $(jq '[.[] | select(.category == "build-error" or .category == "test-failure" or .category == "unknown")] | length' "$analysis_file" 2>/dev/null || echo 0) -eq 0 ]]; }; then + echo "- (none)" + fi + echo "" + + # Run summary + echo "### Run Summary" + echo "" + local total_commits + total_commits=$(cat "$COMMIT_COUNTER_FILE" 2>/dev/null || echo 0) + echo "- **Commits this run:** ${total_commits}" + echo "- **Classification:** deterministic (regex-based)" + echo "" + + echo "---" + echo "*Generated by oape-pr-agent on $(date -u +"%Y-%m-%d %H:%M UTC")*" + } > "$report_file" +} + +# =========================================================================== +# Post Report as PR Comment (update if exists) +# =========================================================================== +post_status_comment() { + local owner="$1" repo="$2" pr_number="$3" + local report_file="${RUNNER_TEMP}/pr-agent-report-${owner}-${repo}-${pr_number}.md" + local state_file="${RUNNER_TEMP}/pr-agent-state-${owner}-${repo}-${pr_number}.json" + local marker="<!-- oape-pr-agent-report -->" + + if [[ ! -f "$report_file" ]]; then + echo "[report] No report file found, skipping comment" >&2 + return 0 + fi + + # Embed state in the comment for cross-run persistence + local state_block="" + if [[ -f "$state_file" ]]; then + local state_b64 + state_b64=$(base64 -w0 < "$state_file") + state_block="<!-- oape-pr-agent-state:${state_b64} -->" + fi + + local body + body="${marker} +${state_block} +$(cat "$report_file")" + + if [[ "$DRY_RUN" == "true" ]]; then + echo "[report] DRY RUN: Would post/update report comment on ${owner}/${repo}#${pr_number}" + audit_log "dry-run" "report" "" "" "would post PR comment" + return 0 + fi + + # Check for existing agent comment + local existing_comment_id + existing_comment_id=$(gh api "repos/${owner}/${repo}/issues/${pr_number}/comments" \ + --jq ".[] | select(.body | contains(\"${marker}\")) | .id" 2>/dev/null | head -1 || true) + + if [[ -n "$existing_comment_id" ]]; then + gh_retry gh api "repos/${owner}/${repo}/issues/comments/${existing_comment_id}" \ + -X PATCH -f body="$body" > /dev/null 2>&1 + echo "[report] Updated existing report comment on ${owner}/${repo}#${pr_number}" + else + gh_retry gh pr comment "$pr_number" --repo "${owner}/${repo}" --body "$body" > /dev/null 2>&1 + echo "[report] Posted new report comment on ${owner}/${repo}#${pr_number}" + fi +} + +# =========================================================================== +# PR Processing Pipeline +# =========================================================================== +process_pr() { + local pr_url="$1" + parse_pr_url "$pr_url" + CURRENT_PR_URL="$pr_url" + + echo "[PR #${PR_NUMBER}] ${OWNER}/${REPO}#${PR_NUMBER} — processing started" + + # Load persisted state from previous runs + load_persisted_state "$OWNER" "$REPO" "$PR_NUMBER" + + # Phase 0: Validate PR is open + local pr_state + pr_state=$(gh_retry gh pr view "$pr_url" --json state -q .state 2>/dev/null || echo "UNKNOWN") + if [[ "$pr_state" != "OPEN" ]]; then + echo "[PR #${PR_NUMBER}] PR is not open (state: ${pr_state}), skipping" + return 0 + fi + + # Phase 0b: Merge conflict check + if ! check_merge_conflicts "$pr_url"; then + generate_status_report "$OWNER" "$REPO" "$PR_NUMBER" "true" + post_status_comment "$OWNER" "$REPO" "$PR_NUMBER" + save_persisted_state "$OWNER" "$REPO" "$PR_NUMBER" + echo "[PR #${PR_NUMBER}] ${OWNER}/${REPO}#${PR_NUMBER} — processing complete (merge conflict)" + return 0 + fi + + # Phase 1: CI Check Monitoring + echo "[PR #${PR_NUMBER}] Phase: CI monitoring — started" + fetch_ci_status "$OWNER" "$REPO" "$PR_NUMBER" + local ci_status + ci_status=$(aggregate_ci_status "$OWNER" "$REPO" "$PR_NUMBER") + echo "[PR #${PR_NUMBER}] Phase: CI monitoring — completed (status: ${ci_status})" + + # Phase 2: Failure Analysis + Auto-Fix (only if there are failures) + if [[ "$ci_status" == "some-failed" ]]; then + echo "[PR #${PR_NUMBER}] Phase: failure analysis — started" + fetch_failure_logs "$OWNER" "$REPO" "$PR_NUMBER" + classify_all_failures "$OWNER" "$REPO" "$PR_NUMBER" + echo "[PR #${PR_NUMBER}] Phase: failure analysis — completed" + + if [[ "$MONITOR_ONLY" != "true" ]]; then + echo "[PR #${PR_NUMBER}] Phase: auto-fix — started" + apply_trivial_fixes "$OWNER" "$REPO" "$PR_NUMBER" || true + echo "[PR #${PR_NUMBER}] Phase: auto-fix — completed" + else + echo "[PR #${PR_NUMBER}] Phase: auto-fix — skipped (monitor-only mode)" + fi + fi + + # Phase 3: Status Report + echo "[PR #${PR_NUMBER}] Phase: status report — started" + generate_status_report "$OWNER" "$REPO" "$PR_NUMBER" + post_status_comment "$OWNER" "$REPO" "$PR_NUMBER" + save_persisted_state "$OWNER" "$REPO" "$PR_NUMBER" + echo "[PR #${PR_NUMBER}] Phase: status report — completed" + + # Clean up log files for this PR to avoid polluting the next PR + rm -f "${RUNNER_TEMP}"/log-*.txt + + echo "[PR #${PR_NUMBER}] ${OWNER}/${REPO}#${PR_NUMBER} — processing complete" +} + +# =========================================================================== +# Timeout Report — posted when per-PR timeout fires +# =========================================================================== +_post_timeout_report() { + local owner="$1" repo="$2" pr_number="$3" + local marker="<!-- oape-pr-agent-report -->" + local body + body="${marker} +## PR Agent Report: ${repo}#${pr_number} + +**Status:** Timed out after ${PR_TIMEOUT_SECONDS}s. The agent could not complete analysis within the per-PR time limit. + +A partial analysis may be available in the [CI run](${OAPE_RUN_URL:-N/A}) logs. + +--- +*Generated by oape-pr-agent on $(date -u +"%Y-%m-%d %H:%M UTC")*" + + if [[ "$DRY_RUN" == "true" ]]; then + echo "[timeout] DRY RUN: Would post timeout report on ${owner}/${repo}#${pr_number}" + return 0 + fi + + local existing_comment_id + existing_comment_id=$(gh api "repos/${owner}/${repo}/issues/${pr_number}/comments" \ + --jq ".[] | select(.body | contains(\"${marker}\")) | .id" 2>/dev/null | head -1 || true) + + if [[ -n "$existing_comment_id" ]]; then + gh_retry gh api "repos/${owner}/${repo}/issues/comments/${existing_comment_id}" \ + -X PATCH -f body="$body" > /dev/null 2>&1 || true + else + gh_retry gh pr comment "$pr_number" --repo "${owner}/${repo}" --body "$body" > /dev/null 2>&1 || true + fi +} + +# =========================================================================== +# Execution Modes +# =========================================================================== +run_periodic() { + echo "[periodic] Starting periodic PR agent run" + + discover_oape_prs + + local pr_count + pr_count=$(wc -l < "$PR_LIST_FILE" 2>/dev/null || echo 0) + pr_count=$((pr_count)) # trim whitespace + + if [[ "$pr_count" -eq 0 ]]; then + echo "[periodic] No open PRs found, nothing to do" + return 0 + fi + + local max_prs="$PR_AGENT_MAX_PRS" + local processed=0 + + while IFS= read -r pr_url; do + [[ -z "$pr_url" ]] && continue + + if [[ "$processed" -ge "$max_prs" ]]; then + echo "[periodic] Reached max PRs (${max_prs}), stopping" + break + fi + + echo "[periodic] Processing PR $((processed + 1))/${max_prs}: ${pr_url}" + + # Build flags for the subprocess invocation + local -a flags=() + [[ "$DRY_RUN" == "true" ]] && flags+=(--dry-run) + [[ "$MONITOR_ONLY" == "true" ]] && flags+=(--monitor-only) + + # Invoke entrypoint.sh as a subprocess per PR, wrapped in timeout. + # Each PR gets its own process for isolation — a crash or hang in one + # does not affect subsequent PRs. + local timeout_rc=0 + timeout --signal=TERM --kill-after=30 "$PR_TIMEOUT_SECONDS" \ + "${BASH_SOURCE[0]}" --mode on-demand --pr-url "$pr_url" "${flags[@]}" \ + || timeout_rc=$? + + if [[ "$timeout_rc" -eq 124 ]]; then + echo "[periodic] PR ${pr_url} — timed out after ${PR_TIMEOUT_SECONDS}s" + parse_pr_url "$pr_url" + _post_timeout_report "$OWNER" "$REPO" "$PR_NUMBER" + audit_log "error" "" "" "" "timeout after ${PR_TIMEOUT_SECONDS}s for ${pr_url}" + elif [[ "$timeout_rc" -ne 0 ]]; then + echo "[periodic] PR ${pr_url} — failed (exit ${timeout_rc}, continuing)" + audit_log "error" "" "" "" "process_pr failed (exit ${timeout_rc}) for ${pr_url}" + else + echo "[periodic] PR ${pr_url} — completed successfully" + fi + + processed=$((processed + 1)) + + # Rate limit between PRs + if [[ "$processed" -lt "$max_prs" ]] && [[ "$processed" -lt "$pr_count" ]]; then + echo "[periodic] Waiting ${RATE_LIMIT_SECONDS}s before next PR..." + sleep "$RATE_LIMIT_SECONDS" + fi + done < "$PR_LIST_FILE" + + echo "[periodic] Processed ${processed} PR(s)" +} + +run_on_demand() { + # Timeout is handled by the outer caller (Prow job timeout, user) for single-PR mode. + local pr_url="$1" + echo "[on-demand] Processing single PR: ${pr_url}" + + parse_pr_url "$pr_url" + validate_repo_allowed "$OWNER" "$REPO" + + process_pr "$pr_url" + echo "[on-demand] Done" +} + +# =========================================================================== +# Main +# =========================================================================== +main() { + echo "============================================" + echo " OAPE PR Lifecycle Agent — Phase 1 MVP" + echo " Mode: ${MODE}" + echo " Monitor Only: ${MONITOR_ONLY}" + echo " Dry Run: ${DRY_RUN}" + echo " Time: $(date -u +"%Y-%m-%d %H:%M UTC")" + echo "============================================" + + run_prechecks + + case "$MODE" in + periodic) + run_periodic + ;; + on-demand) + run_on_demand "$PR_URL_ARG" + ;; + *) + echo "ERROR: Unknown mode: ${MODE} (expected 'periodic' or 'on-demand')" >&2 + exit 1 + ;; + esac + + echo "[main] PR agent run complete" +} + +# Only run main when executed directly (not sourced for timeout subprocess) +if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then + main +fi diff --git a/scripts/pr-agent/log-analyzer.sh b/scripts/pr-agent/log-analyzer.sh new file mode 100755 index 0000000..a02cf8d --- /dev/null +++ b/scripts/pr-agent/log-analyzer.sh @@ -0,0 +1,381 @@ +#!/usr/bin/env bash +# log-analyzer.sh — Claude-powered failure analysis for the OAPE CI Monitor. +# +# Reads CI log files, runs deterministic regex classification first, then +# invokes Claude Code CLI for failures that remain "unknown". Called by +# dispatch.sh for the "investigate" action. +# +# Usage: +# log-analyzer.sh --pr-url <URL> --log-dir <path> [options] +# +# Required: +# --pr-url <URL> PR URL (https://github.com/OWNER/REPO/pull/N) +# --log-dir <path> Directory containing CI log files (log-*.txt) +# +# Optional: +# --job <name> Specific Prow job name to analyze +# --result-file <path> Path to ci-monitor-result.json (for PR context) +# --dry-run Show what would be done without invoking Claude + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +OAPE_ROOT="${OAPE_ROOT:-$(cd "$SCRIPT_DIR/../.." && pwd)}" + +# shellcheck source=scripts/pr-agent/safety.sh +source "${SCRIPT_DIR}/safety.sh" + +# --------------------------------------------------------------------------- +# Configuration +# --------------------------------------------------------------------------- +DRY_RUN="${DRY_RUN:-false}" +MAX_BUDGET_PER_PR="${MAX_BUDGET_PER_PR:-5.00}" +MAX_LOG_LINES="${MAX_LOG_LINES:-1000}" +SKILL_FILE="${OAPE_ROOT}/plugins/oape/skills/ci-monitor/SKILL.md" +CURRENT_PR_URL="" + +# --------------------------------------------------------------------------- +# Usage +# --------------------------------------------------------------------------- +usage() { + echo "Usage: log-analyzer.sh --pr-url <URL> --log-dir <path> [--job <name>] [--result-file <path>] [--dry-run]" + exit 1 +} + +# --------------------------------------------------------------------------- +# Argument parsing +# --------------------------------------------------------------------------- +PR_URL_ARG="" +LOG_DIR="" +JOB_NAME="" +RESULT_FILE="" + +while [[ $# -gt 0 ]]; do + case "$1" in + --pr-url) PR_URL_ARG="$2"; shift 2 ;; + --log-dir) LOG_DIR="$2"; shift 2 ;; + --job) JOB_NAME="$2"; shift 2 ;; + --result-file) RESULT_FILE="$2"; shift 2 ;; + --dry-run) DRY_RUN="true"; shift ;; + --help|-h) usage ;; + *) echo "[log-analyzer] ERROR: Unknown argument: $1" >&2; usage ;; + esac +done + +if [[ -z "$PR_URL_ARG" ]]; then + echo "[log-analyzer] ERROR: --pr-url is required" >&2 + usage +fi + +if [[ -z "$LOG_DIR" ]]; then + echo "[log-analyzer] ERROR: --log-dir is required" >&2 + usage +fi + +# --------------------------------------------------------------------------- +# Parse PR URL +# --------------------------------------------------------------------------- +OWNER="" +REPO="" +PR_NUMBER="" + +if [[ "$PR_URL_ARG" =~ https://github.com/([^/]+)/([^/]+)/pull/([0-9]+) ]]; then + OWNER="${BASH_REMATCH[1]}" + REPO="${BASH_REMATCH[2]}" + PR_NUMBER="${BASH_REMATCH[3]}" +else + echo "[log-analyzer] ERROR: Invalid PR URL format: $PR_URL_ARG" >&2 + exit 1 +fi +CURRENT_PR_URL="https://github.com/${OWNER}/${REPO}/pull/${PR_NUMBER}" + +echo "============================================" +echo " OAPE Log Analyzer" +echo " PR: ${CURRENT_PR_URL}" +echo " Job: ${JOB_NAME:-all}" +echo " Log Dir: ${LOG_DIR}" +echo " Dry Run: ${DRY_RUN}" +echo " Max Budget: \$${MAX_BUDGET_PER_PR}" +echo "============================================" + +# --------------------------------------------------------------------------- +# Collect log files +# --------------------------------------------------------------------------- +declare -a LOG_FILES=() + +if [[ ! -d "$LOG_DIR" ]]; then + echo "[log-analyzer] Log directory does not exist: ${LOG_DIR}" + audit_log "info" "unknown" "" "" "log directory missing: ${LOG_DIR}" + exit 0 +fi + +if [[ -n "$JOB_NAME" ]]; then + log_id=$(echo "$JOB_NAME" | tr '/ ' '__') + target="${LOG_DIR}/log-${log_id}.txt" + if [[ -s "$target" ]]; then + LOG_FILES+=("$target") + fi +fi + +if [[ ${#LOG_FILES[@]} -eq 0 ]]; then + while IFS= read -r -d '' f; do + LOG_FILES+=("$f") + done < <(find "$LOG_DIR" -name 'log-*.txt' -size +0c -print0 2>/dev/null) +fi + +if [[ ${#LOG_FILES[@]} -eq 0 ]]; then + echo "[log-analyzer] No log files found in ${LOG_DIR}" + audit_log "info" "unknown" "" "" "no log files found" + exit 0 +fi + +echo "[log-analyzer] Found ${#LOG_FILES[@]} log file(s)" + +# --------------------------------------------------------------------------- +# Deterministic classification (same regex as auto-fix.sh) +# --------------------------------------------------------------------------- +classify_log() { + local log_file="$1" + local content + content=$(tail -n "$MAX_LOG_LINES" "$log_file") + + # Mode A: Install failure + if echo "$content" | grep -qiE 'level=fatal.*installer|cluster creation failed|bootstrap.*timed out|waiting for bootstrapComplete|cluster-install.*fail'; then + echo "install-failure" + return + fi + + # Mode E: Infrastructure / transient + if echo "$content" | grep -qiE 'ImagePullBackOff|i/o timeout|connection refused|etcdserver: request timed out|lease lost|quota exceeded|InsufficientInstanceCapacity|registry.*timeout|context deadline exceeded'; then + if ! echo "$content" | grep -qiE 'FAIL:.*Test|--- FAIL'; then + echo "infra-flake" + return + fi + fi + + # Mode C: Build / compile failure + if echo "$content" | grep -qiE 'cannot find package|undefined:|syntax error.*\.go|imported and not used|make: \*\*\* .* Error'; then + echo "build-failure" + return + fi + + # Mode D: Lint / static analysis + if echo "$content" | grep -qiE 'golangci-lint|golint|staticcheck|revive|gofmt|goimports|formatting differs|generated code is out of date|make generate|make manifests|deepcopy-gen|boilerplate'; then + echo "lint-failure" + return + fi + + # Mode B: Test failure + if echo "$content" | grep -qiE 'FAIL:.*Test|--- FAIL|FAIL\s+\S+/|test.*failed'; then + echo "test-failure" + return + fi + + echo "unknown" +} + +# --------------------------------------------------------------------------- +# Classify each log file +# --------------------------------------------------------------------------- +declare -A FILE_CLASSIFICATIONS=() +UNKNOWN_FILES=() + +for log_file in "${LOG_FILES[@]}"; do + classification=$(classify_log "$log_file") + FILE_CLASSIFICATIONS["$log_file"]="$classification" + echo "[log-analyzer] ${log_file##*/}: ${classification}" + + if [[ "$classification" == "unknown" ]]; then + UNKNOWN_FILES+=("$log_file") + fi +done + +# --------------------------------------------------------------------------- +# Build analysis output +# --------------------------------------------------------------------------- +ANALYSIS_FILE="${LOG_DIR}/failure-analysis.json" + +build_analysis_entry() { + local job_name="$1" mode="$2" confidence="$3" root_cause="$4" suggested_fix="$5" evidence="$6" + jq -n \ + --arg job "$job_name" \ + --arg mode "$mode" \ + --arg confidence "$confidence" \ + --arg root_cause "$root_cause" \ + --arg suggested_fix "$suggested_fix" \ + --arg evidence "$evidence" \ + '{job: $job, mode: $mode, confidence: $confidence, root_cause: $root_cause, suggested_fix: $suggested_fix, evidence: $evidence}' +} + +ENTRIES="[]" + +for log_file in "${LOG_FILES[@]}"; do + classification="${FILE_CLASSIFICATIONS[$log_file]}" + log_basename="${log_file##*/}" + # shellcheck disable=SC2001 + job_label=$(echo "$log_basename" | sed 's/^log-//;s/\.txt$//') + + if [[ "$classification" != "unknown" ]]; then + entry=$(build_analysis_entry "$job_label" "$classification" "high" \ + "Deterministic classification: ${classification}" \ + "" \ + "Pattern match in ${log_basename}") + ENTRIES=$(echo "$ENTRIES" | jq --argjson e "$entry" '. + [$e]') + fi +done + +# --------------------------------------------------------------------------- +# Claude analysis for unknown failures +# --------------------------------------------------------------------------- +if [[ ${#UNKNOWN_FILES[@]} -gt 0 ]]; then + echo "" + echo "[log-analyzer] ${#UNKNOWN_FILES[@]} file(s) classified as 'unknown' — attempting Claude analysis" + + if [[ "$DRY_RUN" == "true" ]]; then + echo "[log-analyzer] DRY RUN: Would invoke Claude for ${#UNKNOWN_FILES[@]} unknown failure(s)" + for uf in "${UNKNOWN_FILES[@]}"; do + log_basename="${uf##*/}" + # shellcheck disable=SC2001 + job_label=$(echo "$log_basename" | sed 's/^log-//;s/\.txt$//') + entry=$(build_analysis_entry "$job_label" "unknown" "low" \ + "Dry run — Claude analysis not invoked" \ + "" \ + "Would analyze ${log_basename}") + ENTRIES=$(echo "$ENTRIES" | jq --argjson e "$entry" '. + [$e]') + done + audit_log "dry-run" "unknown" "" "" "would invoke Claude for ${#UNKNOWN_FILES[@]} file(s)" + else + # Check Claude CLI availability + CLAUDE_CMD="" + if command -v claude &>/dev/null; then + CLAUDE_CMD="claude" + elif command -v npx &>/dev/null; then + CLAUDE_CMD="npx @anthropic-ai/claude-code" + fi + + if [[ -z "$CLAUDE_CMD" ]]; then + echo "[log-analyzer] Claude CLI not available (install nodejs + npm for npx)" + for uf in "${UNKNOWN_FILES[@]}"; do + log_basename="${uf##*/}" + # shellcheck disable=SC2001 + job_label=$(echo "$log_basename" | sed 's/^log-//;s/\.txt$//') + entry=$(build_analysis_entry "$job_label" "unknown" "low" \ + "Claude CLI not available" \ + "Install nodejs + npm in container image" \ + "") + ENTRIES=$(echo "$ENTRIES" | jq --argjson e "$entry" '. + [$e]') + done + audit_log "skipped" "unknown" "" "" "Claude CLI not available" + else + # Load skill file + SKILL_CONTENT="" + if [[ -f "$SKILL_FILE" ]]; then + SKILL_CONTENT=$(cat "$SKILL_FILE") + else + echo "[log-analyzer] WARN: Skill file not found at ${SKILL_FILE}" + fi + + # Gather PR change context if available + PR_CONTEXT="" + if [[ -n "$RESULT_FILE" && -f "$RESULT_FILE" ]]; then + PR_CONTEXT=$(jq -r '.pr_change_context // empty' "$RESULT_FILE" 2>/dev/null || true) + fi + + for uf in "${UNKNOWN_FILES[@]}"; do + log_basename="${uf##*/}" + # shellcheck disable=SC2001 + job_label=$(echo "$log_basename" | sed 's/^log-//;s/\.txt$//') + + echo "[log-analyzer] Analyzing ${log_basename} with Claude..." + + log_excerpt=$(tail -n "$MAX_LOG_LINES" "$uf") + + prompt="You are analyzing a CI failure for an OpenShift operator PR. + +${SKILL_CONTENT:+## CI Monitor Skill Reference +$SKILL_CONTENT + +--- +} +## Task + +Analyze the following CI log for the failure mode and root cause. + +Job: ${job_label} +PR: ${CURRENT_PR_URL} +Repository: ${OWNER}/${REPO} +${PR_CONTEXT:+ +PR Change Context: +$PR_CONTEXT +} +## CI Log (last ${MAX_LOG_LINES} lines) + +\`\`\` +${log_excerpt} +\`\`\` + +## Required Output + +Respond with ONLY a JSON object (no markdown fencing, no explanation) with these fields: +- \"mode\": one of \"install-failure\", \"test-failure\", \"build-failure\", \"lint-failure\", \"infra-flake\", \"unknown\" +- \"root_cause\": one-sentence root cause description +- \"confidence\": one of \"high\", \"medium\", \"low\" +- \"suggested_fix\": actionable fix suggestion (or empty string if none) +- \"evidence\": key log line(s) supporting the classification" + + claude_output="" + if claude_output=$($CLAUDE_CMD --print \ + --max-turns 1 \ + --max-budget-usd "$MAX_BUDGET_PER_PR" \ + -p "$prompt" 2>&1); then + + # Try to parse Claude's response as JSON + parsed="" + if parsed=$(echo "$claude_output" | grep -oP '\{[^{}]*\}' | head -1 | jq '.' 2>/dev/null); then + mode=$(echo "$parsed" | jq -r '.mode // "unknown"') + root_cause=$(echo "$parsed" | jq -r '.root_cause // "Claude analysis"') + confidence=$(echo "$parsed" | jq -r '.confidence // "medium"') + suggested_fix=$(echo "$parsed" | jq -r '.suggested_fix // ""') + evidence=$(echo "$parsed" | jq -r '.evidence // ""') + + echo "[log-analyzer] Claude result for ${job_label}: mode=${mode}, confidence=${confidence}" + entry=$(build_analysis_entry "$job_label" "$mode" "$confidence" "$root_cause" "$suggested_fix" "$evidence") + ENTRIES=$(echo "$ENTRIES" | jq --argjson e "$entry" '. + [$e]') + audit_log "analyzed" "$mode" "" "" "Claude analysis: ${root_cause}" + else + echo "[log-analyzer] WARN: Could not parse Claude output as JSON for ${job_label}" + entry=$(build_analysis_entry "$job_label" "unknown" "low" \ + "Claude output not parseable" \ + "" \ + "Raw output saved to log") + ENTRIES=$(echo "$ENTRIES" | jq --argjson e "$entry" '. + [$e]') + audit_log "error" "unknown" "" "" "Claude output not parseable for ${job_label}" + fi + else + echo "[log-analyzer] WARN: Claude invocation failed for ${job_label}" + entry=$(build_analysis_entry "$job_label" "unknown" "low" \ + "Claude invocation failed" \ + "" \ + "") + ENTRIES=$(echo "$ENTRIES" | jq --argjson e "$entry" '. + [$e]') + audit_log "error" "unknown" "" "" "Claude invocation failed for ${job_label}" + fi + done + fi + fi +else + echo "[log-analyzer] All failures classified deterministically — no Claude analysis needed" +fi + +# --------------------------------------------------------------------------- +# Write analysis output +# --------------------------------------------------------------------------- +echo "$ENTRIES" | jq '{analysis: ., pr_url: $pr, analyzed_at: $ts}' \ + --arg pr "$CURRENT_PR_URL" \ + --arg ts "$(date -u +"%Y-%m-%dT%H:%M:%SZ")" \ + > "$ANALYSIS_FILE" + +echo "" +echo "[log-analyzer] Analysis written to ${ANALYSIS_FILE}" +echo "[log-analyzer] Summary:" +echo "$ENTRIES" | jq -r '.[] | " \(.job): \(.mode) (\(.confidence))"' diff --git a/scripts/pr-agent/safety.sh b/scripts/pr-agent/safety.sh new file mode 100755 index 0000000..f43262c --- /dev/null +++ b/scripts/pr-agent/safety.sh @@ -0,0 +1,173 @@ +#!/usr/bin/env bash +# safety.sh — Sourced utility library providing shared guardrail functions +# for the OAPE PR Lifecycle Agent. Source this file; do not execute directly. +# +# Usage: source scripts/pr-agent/safety.sh + +# Guard against direct execution +if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then + echo "ERROR: safety.sh must be sourced, not executed directly" >&2 + exit 1 +fi + +# --------------------------------------------------------------------------- +# Configuration (overridable via environment) +# --------------------------------------------------------------------------- +MAX_COMMITS_PER_RUN="${MAX_COMMITS_PER_RUN:-10}" +MAX_COMMITS_PER_PR="${MAX_COMMITS_PER_PR:-3}" +MAX_DIFF_LINES="${MAX_DIFF_LINES:-500}" +COMMIT_COUNTER_FILE="${RUNNER_TEMP:-/tmp}/pr-agent-commit-count.txt" +AUDIT_LOG="${RUNNER_TEMP:-/tmp}/pr-agent-audit-${GITHUB_RUN_ID:-${BUILD_ID:-local}}.jsonl" + +# --------------------------------------------------------------------------- +# File blocklist patterns +# --------------------------------------------------------------------------- +# Protect actual secret storage files, CI/container configs, RBAC manifests, +# and dependency lock files. Go source files that operate on Kubernetes +# Secret/Token resources are NOT blocked — only files that store secrets. + +# Default patterns (go.mod and go.sum blocked) +BLOCKED_PATTERNS='\.(key|pem|crt|cert|p12|pfx)$' +BLOCKED_PATTERNS+='|\.env$' +BLOCKED_PATTERNS+='|credentials\.' +BLOCKED_PATTERNS+='|(^|/)kubeconfig$' +BLOCKED_PATTERNS+='|(^|/)Dockerfile$|(^|/)Containerfile$|\.dockerignore$' +BLOCKED_PATTERNS+='|\.github/workflows|\.tekton/' +BLOCKED_PATTERNS+='|(^|/)Makefile$' +BLOCKED_PATTERNS+='|rbac/.*\.yaml|clusterrole.*\.yaml' +BLOCKED_PATTERNS+='|go\.mod$|go\.sum$' + +# Relaxed patterns for trivial-generated-files (go.mod/go.sum allowed +# because make generate legitimately runs go mod tidy) +BLOCKED_PATTERNS_GENERATED='\.(key|pem|crt|cert|p12|pfx)$' +BLOCKED_PATTERNS_GENERATED+='|\.env$' +BLOCKED_PATTERNS_GENERATED+='|credentials\.' +BLOCKED_PATTERNS_GENERATED+='|(^|/)kubeconfig$' +BLOCKED_PATTERNS_GENERATED+='|(^|/)Dockerfile$|(^|/)Containerfile$|\.dockerignore$' +BLOCKED_PATTERNS_GENERATED+='|\.github/workflows|\.tekton/' +BLOCKED_PATTERNS_GENERATED+='|(^|/)Makefile$' +BLOCKED_PATTERNS_GENERATED+='|rbac/.*\.yaml|clusterrole.*\.yaml' + +# --------------------------------------------------------------------------- +# check_blocklist — returns 0 (safe) or 1 (blocked) +# $1: newline-separated file paths to check +# $2: (optional) failure category — "trivial-generated-files" relaxes go.mod/go.sum +# --------------------------------------------------------------------------- +check_blocklist() { + local files="$1" + local category="${2:-}" + local patterns="$BLOCKED_PATTERNS" + + if [[ "$category" == "trivial-generated-files" ]]; then + patterns="$BLOCKED_PATTERNS_GENERATED" + fi + + if [[ -z "$files" ]]; then + return 0 + fi + + if echo "$files" | grep -qE "$patterns"; then + return 1 + fi + return 0 +} + +# --------------------------------------------------------------------------- +# audit_log — append a structured JSONL entry +# $1: action (auto-fix, blocked, skipped, reverted, dry-run, error, info) +# $2: category (trivial-format, trivial-generated-files, etc.) +# $3: files (space-separated list) +# $4: commit (SHA or empty) +# $5: outcome (human-readable description) +# --------------------------------------------------------------------------- +audit_log() { + local action="${1:-}" category="${2:-}" files="${3:-}" commit="${4:-}" outcome="${5:-}" + local ts + ts=$(date -u +"%Y-%m-%dT%H:%M:%SZ") + local files_json + files_json=$(echo "$files" | tr ' ' '\n' | jq -R -s 'split("\n") | map(select(. != ""))' 2>/dev/null || echo '[]') + + printf '{"ts":"%s","pr":"%s","action":"%s","type":"%s","files":%s,"commit":"%s","outcome":"%s"}\n' \ + "$ts" "${CURRENT_PR_URL:-}" "$action" "$category" \ + "$files_json" "$commit" "$outcome" \ + >> "$AUDIT_LOG" +} + +# --------------------------------------------------------------------------- +# check_commit_limit — returns 0 (within limits) or 1 (limit reached) +# $1: per-PR commit count for the current PR +# --------------------------------------------------------------------------- +check_commit_limit() { + local pr_commits="${1:-0}" + local total_commits + total_commits=$(cat "$COMMIT_COUNTER_FILE" 2>/dev/null || echo 0) + + if [[ "$total_commits" -ge "$MAX_COMMITS_PER_RUN" ]]; then + echo "GUARDRAIL: Run commit limit reached (${total_commits}/${MAX_COMMITS_PER_RUN})" >&2 + return 1 + fi + if [[ "$pr_commits" -ge "$MAX_COMMITS_PER_PR" ]]; then + echo "GUARDRAIL: Per-PR commit limit reached (${pr_commits}/${MAX_COMMITS_PER_PR})" >&2 + return 1 + fi + return 0 +} + +# --------------------------------------------------------------------------- +# increment_commit_count — bump the shared counter file by 1, echo new total +# --------------------------------------------------------------------------- +increment_commit_count() { + local total + total=$(cat "$COMMIT_COUNTER_FILE" 2>/dev/null || echo 0) + total=$((total + 1)) + echo "$total" > "$COMMIT_COUNTER_FILE" + echo "$total" +} + +# --------------------------------------------------------------------------- +# check_diff_size — returns 0 (within limit) or 1 (too large) +# Checks staged + unstaged changes against MAX_DIFF_LINES. +# --------------------------------------------------------------------------- +check_diff_size() { + local diff_lines + diff_lines=$(git diff --numstat | awk '{s+=$1+$2} END {print s+0}') + # Include untracked files that would be staged + local untracked_lines + untracked_lines=$(git ls-files --others --exclude-standard -z 2>/dev/null \ + | xargs -0 wc -l 2>/dev/null | tail -1 | awk '{print $1+0}' || echo 0) + diff_lines=$((diff_lines + untracked_lines)) + + if [[ "$diff_lines" -gt "$MAX_DIFF_LINES" ]]; then + echo "GUARDRAIL: Diff too large (${diff_lines} lines > ${MAX_DIFF_LINES} limit)" >&2 + return 1 + fi + return 0 +} + +# --------------------------------------------------------------------------- +# gh_retry — retry a command with exponential backoff +# All arguments are passed through as the command to execute. +# Retries 3 times at 5s / 15s / 45s intervals. +# --------------------------------------------------------------------------- +gh_retry() { + local retries=3 delay=5 + for ((i = 1; i <= retries; i++)); do + if "$@"; then + return 0 + fi + if [[ "$i" -lt "$retries" ]]; then + echo "[retry] Attempt ${i}/${retries} failed, waiting ${delay}s..." >&2 + sleep "$delay" + delay=$((delay * 3)) + fi + done + echo "[retry] All ${retries} attempts failed for: $*" >&2 + return 1 +} + +# --------------------------------------------------------------------------- +# Initialize commit counter file if it doesn't exist +# --------------------------------------------------------------------------- +if [[ ! -f "$COMMIT_COUNTER_FILE" ]]; then + echo 0 > "$COMMIT_COUNTER_FILE" +fi diff --git a/scripts/pr-agent/test-dry-run.sh b/scripts/pr-agent/test-dry-run.sh new file mode 100755 index 0000000..9fc7506 --- /dev/null +++ b/scripts/pr-agent/test-dry-run.sh @@ -0,0 +1,250 @@ +#!/usr/bin/env bash +# test-dry-run.sh — Validation suite for the OAPE PR agent scripts. +# +# Runs: +# 1. shellcheck on all agent + monitor scripts +# 2. Dry-run integration test against a real PR +# 3. Output file verification +# +# Usage: +# scripts/pr-agent/test-dry-run.sh [--pr-url <URL>] +# +# Environment: +# GH_TOKEN — required for GitHub API access +# TEST_PR_URL — override the default test PR (optional) +# RUNNER_TEMP — temp directory (default: mktemp -d) + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" + +PASS_COUNT=0 +FAIL_COUNT=0 +SKIP_COUNT=0 + +_pass() { echo " PASS: $1"; PASS_COUNT=$((PASS_COUNT + 1)); } +_fail() { echo " FAIL: $1" >&2; FAIL_COUNT=$((FAIL_COUNT + 1)); } +_skip() { echo " SKIP: $1"; SKIP_COUNT=$((SKIP_COUNT + 1)); } + +# Parse arguments +TEST_PR_URL="${TEST_PR_URL:-}" +while [[ $# -gt 0 ]]; do + case "$1" in + --pr-url) TEST_PR_URL="$2"; shift 2 ;; + *) shift ;; + esac +done + +echo "============================================" +echo " OAPE PR Agent — Validation Suite" +echo " Time: $(date -u +"%Y-%m-%d %H:%M UTC")" +echo "============================================" +echo "" + +# ========================================================================= +# Phase 1: Shellcheck +# ========================================================================= +echo "=== Phase 1: shellcheck ===" + +SCRIPTS=( + "${REPO_ROOT}/scripts/pr-agent/entrypoint.sh" + "${REPO_ROOT}/scripts/pr-agent/safety.sh" + "${REPO_ROOT}/scripts/pr-agent/auto-fix.sh" + "${REPO_ROOT}/scripts/pr-agent/log-analyzer.sh" + "${REPO_ROOT}/scripts/ci-monitor/monitor.sh" + "${REPO_ROOT}/scripts/ci-monitor/dispatch.sh" +) + +if ! command -v shellcheck &>/dev/null; then + _skip "shellcheck is not installed (dnf install ShellCheck)" +else + shellcheck_ok=true + for script in "${SCRIPTS[@]}"; do + local_name="${script#"${REPO_ROOT}/"}" + if shellcheck -x -s bash "$script" 2>/dev/null; then + _pass "${local_name}" + else + _fail "${local_name}" + shellcheck_ok=false + fi + done + + if [[ "$shellcheck_ok" == "true" ]]; then + echo " All scripts are shellcheck-clean" + fi +fi + +echo "" + +# ========================================================================= +# Phase 2: Dry-run integration test +# ========================================================================= +echo "=== Phase 2: Dry-run integration test ===" + +if [[ -z "$TEST_PR_URL" ]]; then + echo " No --pr-url provided, attempting to find an open PR on an allowed repo..." + # Pick the first repo from team-repos.csv and find an open PR + if [[ -f "${REPO_ROOT}/deploy/config/team-repos.csv" ]]; then + while IFS=, read -r _product _role repo_url; do + local_repo=$(echo "$repo_url" | sed 's|https://github.com/||;s|\.git$||') + pr_url=$(gh pr list --repo "$local_repo" --state open --json url --limit 1 -q '.[0].url' 2>/dev/null || true) + if [[ -n "$pr_url" ]]; then + TEST_PR_URL="$pr_url" + break + fi + done < <(tail -n +2 "${REPO_ROOT}/deploy/config/team-repos.csv") + fi +fi + +if [[ -z "$TEST_PR_URL" ]]; then + _skip "No test PR URL available (pass --pr-url or set TEST_PR_URL)" + echo "" + echo "=== Phase 2b: auto-fix.sh validation ===" + _skip "Skipped (no test PR URL)" + echo "" + echo "=== Phase 3: Output verification ===" + _skip "Skipped (no integration test ran)" +else + echo " Test PR: ${TEST_PR_URL}" + + # Extract owner/repo/number for output file verification + if [[ "$TEST_PR_URL" =~ https://github.com/([^/]+)/([^/]+)/pull/([0-9]+) ]]; then + TEST_OWNER="${BASH_REMATCH[1]}" + TEST_REPO="${BASH_REMATCH[2]}" + TEST_PR_NUMBER="${BASH_REMATCH[3]}" + else + _fail "Invalid test PR URL format: ${TEST_PR_URL}" + echo "" + echo "============================================" + echo " Results: ${PASS_COUNT} passed, ${FAIL_COUNT} failed, ${SKIP_COUNT} skipped" + echo "============================================" + exit 1 + fi + + export DRY_RUN=true + export MONITOR_ONLY=true + export RUNNER_TEMP + RUNNER_TEMP=$(mktemp -d) + export GH_TOKEN="${GH_TOKEN:-$(gh auth token 2>/dev/null || echo '')}" + + echo " DRY_RUN=true, MONITOR_ONLY=true" + echo " RUNNER_TEMP=${RUNNER_TEMP}" + + # Run the agent + if "${REPO_ROOT}/scripts/pr-agent/entrypoint.sh" \ + --mode on-demand --pr-url "$TEST_PR_URL" --dry-run --monitor-only; then + _pass "entrypoint.sh exited successfully" + else + _fail "entrypoint.sh exited with non-zero status" + fi + + echo "" + + # ========================================================================= + # Phase 2b: auto-fix.sh dry-run validation + # ========================================================================= + echo "=== Phase 2b: auto-fix.sh dry-run test ===" + + for test_cat in trivial-format trivial-import trivial-lint trivial-generated-files; do + echo " Testing category: ${test_cat}" + if "${REPO_ROOT}/scripts/pr-agent/auto-fix.sh" \ + --pr-url "$TEST_PR_URL" --category "$test_cat" --dry-run 2>&1 | grep -q "DRY RUN\|No changes\|not available"; then + _pass "auto-fix.sh --category ${test_cat} --dry-run" + else + _fail "auto-fix.sh --category ${test_cat} --dry-run" + fi + done + + # Test lint-failure coarse category (triggers fine-grained refinement) + echo " Testing coarse category: lint-failure" + if "${REPO_ROOT}/scripts/pr-agent/auto-fix.sh" \ + --pr-url "$TEST_PR_URL" --category lint-failure --dry-run 2>&1 | grep -q "DRY RUN\|No changes\|Refined\|not available"; then + _pass "auto-fix.sh --category lint-failure --dry-run (fine-grained refinement)" + else + _fail "auto-fix.sh --category lint-failure --dry-run" + fi + + echo "" + + # ========================================================================= + # Phase 2c: log-analyzer.sh dry-run validation + # ========================================================================= + echo "=== Phase 2c: log-analyzer.sh dry-run test ===" + + LOG_TEST_DIR=$(mktemp -d) + echo 'pkg/foo.go:10: undefined: Bar' > "${LOG_TEST_DIR}/log-test-job.txt" + echo 'make: *** [build] Error 2' >> "${LOG_TEST_DIR}/log-test-job.txt" + + echo " Testing with synthetic build-failure log" + if "${REPO_ROOT}/scripts/pr-agent/log-analyzer.sh" \ + --pr-url "$TEST_PR_URL" --log-dir "$LOG_TEST_DIR" --dry-run 2>&1 | grep -q "build-failure\|DRY RUN\|Analysis written"; then + _pass "log-analyzer.sh --dry-run (deterministic classification)" + else + _fail "log-analyzer.sh --dry-run" + fi + + echo " Testing with empty log directory" + EMPTY_LOG_DIR=$(mktemp -d) + if "${REPO_ROOT}/scripts/pr-agent/log-analyzer.sh" \ + --pr-url "$TEST_PR_URL" --log-dir "$EMPTY_LOG_DIR" --dry-run 2>&1 | grep -q "No log files"; then + _pass "log-analyzer.sh --dry-run (empty dir — graceful exit)" + else + _fail "log-analyzer.sh --dry-run (empty dir)" + fi + + rm -rf "$LOG_TEST_DIR" "$EMPTY_LOG_DIR" + + echo "" + + # ========================================================================= + # Phase 3: Output verification + # ========================================================================= + echo "=== Phase 3: Output verification ===" + + # Verify CI status JSON + CI_STATUS_FILE="${RUNNER_TEMP}/ci-status-${TEST_OWNER}-${TEST_REPO}-${TEST_PR_NUMBER}.json" + if [[ -f "$CI_STATUS_FILE" ]]; then + _pass "CI status file exists: $(basename "$CI_STATUS_FILE")" + if jq empty "$CI_STATUS_FILE" 2>/dev/null; then + _pass "CI status file is valid JSON" + else + _fail "CI status file is not valid JSON" + fi + else + _fail "CI status file not generated: ci-status-${TEST_OWNER}-${TEST_REPO}-${TEST_PR_NUMBER}.json" + fi + + # Verify report + REPORT_FILE="${RUNNER_TEMP}/pr-agent-report-${TEST_OWNER}-${TEST_REPO}-${TEST_PR_NUMBER}.md" + if [[ -f "$REPORT_FILE" ]]; then + _pass "Report file exists: $(basename "$REPORT_FILE")" + if grep -q "## PR Agent Report" "$REPORT_FILE"; then + _pass "Report contains expected header" + else + _fail "Report missing '## PR Agent Report' header" + fi + else + _fail "Report file not generated: pr-agent-report-${TEST_OWNER}-${TEST_REPO}-${TEST_PR_NUMBER}.md" + fi + + # Verify state file + STATE_FILE="${RUNNER_TEMP}/pr-agent-state-${TEST_OWNER}-${TEST_REPO}-${TEST_PR_NUMBER}.json" + if [[ -f "$STATE_FILE" ]]; then + _pass "State file exists: $(basename "$STATE_FILE")" + else + _skip "State file not generated (may be normal for first run on this PR)" + fi + + # Clean up + rm -rf "$RUNNER_TEMP" +fi + +echo "" +echo "============================================" +echo " Results: ${PASS_COUNT} passed, ${FAIL_COUNT} failed, ${SKIP_COUNT} skipped" +echo "============================================" + +if [[ "$FAIL_COUNT" -gt 0 ]]; then + exit 1 +fi