Ci monitor phase 2

AGENTS.md

@@ -21,7 +21,8 @@ This project provides AI-driven tools for end-to-end feature development in Open
 | `/oape:e2e-generate <base-branch>`                               | Generate e2e test artifacts from git diff against base branch  |
 | `/oape:predict-regressions <base-branch>`                        | Predict API regressions and breaking changes from git diff     |
 | `/oape:review <ticket_id> [base_ref]`                            | Production-grade code review against Jira requirements         |
-| `/oape:implement-review-fixes <report>`                          | Automatically apply fixes from a review report                 |
+| `/oape:implement-review-fixes <review_report_json>`              | Automatically apply fixes from a review report                 |
+| `/oape:pr-agent <PR-URL> [--dry-run] [--monitor-only]`           | Monitor PR CI status, classify failures, generate report       |
 
 ### Input Sources for api-generate and api-implement
 
@@ -102,7 +103,7 @@ When using a design document (gist), it should contain structured implementation
 
 ## Supported Operator Repositories
 
-The allowed repositories and their base branches are defined in [`team-repos.csv`](config/team-repos.csv). DO NOT raise PRs on any repos beyond that list. Always read `team-repos.csv` to determine the correct repo URL and base branch before cloning or creating branches.
+The allowed repositories and their base branches are defined in [`team-repos.csv`](deploy/config/team-repos.csv). DO NOT raise PRs on any repos beyond that list. Always read `team-repos.csv` to determine the correct repo URL and base branch before cloning or creating branches.
 
 ---
 
@@ -119,7 +120,19 @@ The commands automatically detect which framework the repository uses:
 
 ## Project Structure
 
-
+```
+scripts/ci-monitor/       CI monitoring pipeline (monitor.sh, dispatch.sh)
+scripts/pr-agent/         PR lifecycle agent (entrypoint, auto-fix, safety, test-dry-run)
+plugins/oape/commands/    Claude Code command definitions (/oape:* commands)
+plugins/oape/skills/      Claude Code skill definitions (injected into agent prompts)
+config/                   Tool configuration (config.json)
+images/                   Container image Dockerfiles (ci-monitor, agent-worker, go-server, gh-token-minter)
+docs/                     Architecture docs, Prow config templates
+deploy/                   Kubernetes deployment manifests
+go-server/                Go HTTP server for agent job execution
+agent/                    Python agent wrapper
+gh-token-minter/          GitHub App token generation utility
+```
 
 ---
 

README.md

@@ -64,7 +64,7 @@ The following tools must be installed and available on your system:
 - **GitHub CLI (`gh`)**: [cli.github.com](https://cli.github.com/)
 - **make**: eg. `make generate`, `make build`, etc.
 
-The container image used in [Dockerfile](./Dockerfile) already adds these dependencies, but you need to manage the credentials inside the container.
+The container images in [images/](./images/) already add these dependencies, but you need to manage the credentials inside the container.
 
 ### Optional
 
@@ -113,16 +113,16 @@ ln -s oape-ai-e2e ~/.cursor/commands/oape-ai-e2e
 
 | Plugin | Description | Commands |
 | ------------------------- | ---------------------------------------------- | --------------------------------------------------------------------------- |
-| **[oape](plugins/oape/)** | AI-driven OpenShift operator development tools | `/oape:init`, `/oape:api-generate`, `/oape:api-generate-tests`, `/oape:api-implement`, `/oape:analyze-rfe`, `/oape:e2e-generate`, `/oape:predict-regressions`, `/oape:review`, `/oape:implement-review-fixes` |
+| **[oape](plugins/oape/)** | AI-driven OpenShift operator development tools | `/oape:init`, `/oape:api-generate`, `/oape:api-generate-tests`, `/oape:api-implement`, `/oape:analyze-rfe`, `/oape:e2e-generate`, `/oape:predict-regressions`, `/oape:review`, `/oape:implement-review-fixes`, `/oape:pr-agent` |
 
 ## Commands
 
 ### `/oape:init` -- Clone an Operator Repository
 
-Clones an allowed OpenShift operator repository by short name into the current directory.
+Clones an allowed OpenShift operator repository into the current directory and checks out the specified base branch.
 
 ```shell
-/oape:init cert-manager-operator
+/oape:init https://github.com/openshift/cert-manager-operator main
 ```
 
 ### `/oape:api-generate` -- Generate API Types from Enhancement Proposal
@@ -189,13 +189,23 @@ Performs a production-grade code review that verifies code changes against Jira
 Automatically applies code fixes from a review report.
 
 ```shell
-/oape:implement-review-fixes <report-path>
+/oape:implement-review-fixes <review_report_json>
+```
+
+### `/oape:pr-agent` -- Monitor PR CI Status and Triage Failures
+
+Monitors a PR's CI checks, classifies failures deterministically, and posts a structured Markdown report as a PR comment.
+
+```shell
+/oape:pr-agent https://github.com/openshift/cert-manager-operator/pull/123
+/oape:pr-agent https://github.com/openshift/cert-manager-operator/pull/123 --dry-run
+/oape:pr-agent https://github.com/openshift/cert-manager-operator/pull/123 --monitor-only
 ```
 
 **Typical workflow:**
 ```shell
 # Step 1: Clone the operator repository
-/oape:init cert-manager-operator
+/oape:init https://github.com/openshift/cert-manager-operator main
 
 # Step 2: Generate API types
 /oape:api-generate https://github.com/openshift/enhancements/pull/1234
@@ -219,10 +229,12 @@ Automatically applies code fixes from a review report.
 
 ```shell
 podman build -t quay.io/your-username/oape-ai:agent-worker -f images/agent-worker.Dockerfile .
+podman build -t quay.io/your-username/oape-ai:ci-monitor -f images/ci-monitor.Dockerfile .
 podman build -t quay.io/your-username/oape-ai:gh-token-minter -f images/gh-token-minter.Dockerfile .
 podman build -t quay.io/your-username/oape-ai:go-server -f images/go-server.Dockerfile .
 
 podman push quay.io/your-username/oape-ai:agent-worker
+podman push quay.io/your-username/oape-ai:ci-monitor
 podman push quay.io/your-username/oape-ai:gh-token-minter
 podman push quay.io/your-username/oape-ai:go-server
 ```

docs/prow-ci-operator-config.yaml

@@ -0,0 +1,177 @@
+  # oape-ci-monitor — Prow presubmit job for CI failure analysis.
+#
+# Add the snippets below to the target repo's ci-operator config in
+# openshift/release. Replace REPO_ORG, REPO_NAME, and BRANCH as needed.
+#
+# File: ci-operator/config/REPO_ORG/REPO_NAME/REPO_ORG-REPO_NAME-BRANCH.yaml
+#
+# The job runs as an optional, always-run presubmit alongside other CI jobs.
+# It polls until all other checks reach a terminal state, then:
+#   1. Collects Prow build logs from GCS via gcsweb
+#   2. Classifies failures deterministically (regex-based, zero API cost)
+#   3. Queries Sippy for flake history
+#   4. Posts a structured analysis report as a PR comment
+#
+# Trigger manually: /test oape-ci-monitor
+#
+# Prerequisites:
+#   Secrets in test-credentials namespace:
+#     - oap-lts-claude-gcp-vertex-sa       (GCP ADC for Vertex AI)
+#     - openshift-app-platform-shift-github-bot  (GitHub App ID + private key)
+#
+# Auth strategy:
+#   Phase 1 (report-only): Tries GitHub App token first. If the App is not
+#   installed on the target repo, falls back to GITHUB_TOKEN for read + comment.
+#   Phase 2+ (auto-fix): Requires the GitHub App token for pushes that trigger CI
+#   (GITHUB_TOKEN pushes don't trigger downstream CI due to anti-recursion).
+
+# ─────────────────────────────────────────────────────────────────────
+# 1. Add inline image build under images.items[]
+# ─────────────────────────────────────────────────────────────────────
+#
+# images:
+#   items:
+#   - dockerfile_literal: |-
+#       FROM registry.access.redhat.com/ubi9/go-toolset
+#       USER 0
+#       RUN dnf install -y git make jq && \
+#           dnf install -y 'dnf-command(config-manager)' && \
+#           dnf config-manager --add-repo https://cli.github.com/packages/rpm/gh-cli.repo && \
+#           dnf install -y gh && \
+#           dnf clean all
+#       WORKDIR /app
+#       RUN git clone --depth 1 -b OAPE-752 https://github.com/openshift-eng/oape-ai-e2e.git /tmp/oape && \
+#           cp -r /tmp/oape/scripts /app/scripts && \
+#           cp -r /tmp/oape/plugins /plugins && \
+#           mkdir -p /config && cp -r /tmp/oape/deploy/config/* /config/ && \
+#           rm -rf /tmp/oape
+#       RUN go install golang.org/x/tools/cmd/goimports@latest && \
+#           curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b /usr/local/bin
+#       RUN dnf install -y nodejs npm && dnf clean all
+#       RUN git config --global user.name "openshift-app-platform-shift-bot" && \
+#           git config --global user.email "267347085+openshift-app-platform-shift-bot@users.noreply.github.com"
+#       RUN chmod -R g=u /opt/app-root/src
+#       USER 1001
+#     to: ci-monitor-agent
+
+# ─────────────────────────────────────────────────────────────────────
+# 2. Exclude from promotion (add under promotion.to[].excluded_images)
+# ─────────────────────────────────────────────────────────────────────
+#
+# promotion:
+#   to:
+#   - excluded_images:
+#     - ci-monitor-agent
+#     name: "5.0"
+#     namespace: ocp
+
+# ─────────────────────────────────────────────────────────────────────
+# 3. Add presubmit test under tests[]
+# ─────────────────────────────────────────────────────────────────────
+- always_run: true
+  as: oape-ci-monitor
+  optional: true
+  steps:
+    test:
+    - as: monitor
+      commands: |
+        set -euo pipefail
+
+        echo "[setup] Starting oape-ci-monitor for ${REPO_OWNER}/${REPO_NAME} PR#${PULL_NUMBER}"
+
+        # --- Rehearsal detection ---
+        # Prow rehearsal runs against openshift/release, not the target repo.
+        # Switch to a real target-repo PR to validate the full pipeline.
+        if [[ "${REPO_NAME}" == "release" && "${REPO_OWNER}" == "openshift" ]]; then
+          echo "[setup] Detected openshift/release context — switching to test target"
+          export REPO_OWNER="REPO_ORG"
+          export REPO_NAME="REPO_NAME"
+          TEST_PR=$(curl -s "https://api.github.com/repos/${REPO_OWNER}/${REPO_NAME}/pulls?state=open&per_page=1" \
+            | python3 -c "import sys,json; data=json.load(sys.stdin); print(data[0]['number'] if data else '')" 2>/dev/null || echo "")
+          if [[ -z "$TEST_PR" ]]; then
+            echo "[setup] No open PRs found on ${REPO_OWNER}/${REPO_NAME} — skipping"
+            exit 0
+          fi
+          export PULL_NUMBER="$TEST_PR"
+          export PR_URL="https://github.com/${REPO_OWNER}/${REPO_NAME}/pull/${PULL_NUMBER}"
+          echo "[setup] Testing against ${REPO_OWNER}/${REPO_NAME}#${PULL_NUMBER}"
+        fi
+
+        # --- GitHub auth: try App token, fall back to GITHUB_TOKEN ---
+        # App token is preferred (required for Phase 2+ pushes that trigger CI).
+        # For Phase 1 (report-only), GITHUB_TOKEN is sufficient for read + comment.
+        USE_APP_TOKEN="false"
+        if [[ -f /var/run/github-app/app-id && -f /var/run/github-app/private-key.pem ]]; then
+          echo "[auth] Attempting GitHub App token..."
+          APP_ID=$(cat /var/run/github-app/app-id)
+          PEM_PATH="/var/run/github-app/private-key.pem"
+          HEADER=$(printf '{"alg":"RS256","typ":"JWT"}' | openssl base64 -e -A | tr '+/' '-_' | tr -d '=')
+          NOW=$(date +%s); EXP=$((NOW + 300))
+          PAYLOAD=$(printf '{"iat":%d,"exp":%d,"iss":"%s"}' "$NOW" "$EXP" "$APP_ID" | openssl base64 -e -A | tr '+/' '-_' | tr -d '=')
+          SIGNATURE=$(printf '%s' "${HEADER}.${PAYLOAD}" | openssl dgst -sha256 -sign "$PEM_PATH" -binary | openssl base64 -e -A | tr '+/' '-_' | tr -d '=')
+          JWT="${HEADER}.${PAYLOAD}.${SIGNATURE}"
+
+          INSTALL_RESPONSE=$(curl -s -w "\n%{http_code}" -H "Authorization: Bearer ${JWT}" -H "Accept: application/vnd.github+json" \
+            "https://api.github.com/repos/${REPO_OWNER}/${REPO_NAME}/installation")
+          HTTP_CODE=$(echo "$INSTALL_RESPONSE" | tail -1)
+          INSTALL_BODY=$(echo "$INSTALL_RESPONSE" | sed '$d')
+
+          if [[ "$HTTP_CODE" -eq 200 ]]; then
+            INST_ID=$(echo "$INSTALL_BODY" | python3 -c "import sys,json; print(json.load(sys.stdin)['id'])")
+            TOKEN_RESPONSE=$(curl -s -w "\n%{http_code}" -X POST -H "Authorization: Bearer ${JWT}" -H "Accept: application/vnd.github+json" \
+              "https://api.github.com/app/installations/${INST_ID}/access_tokens")
+            T_CODE=$(echo "$TOKEN_RESPONSE" | tail -1)
+            T_BODY=$(echo "$TOKEN_RESPONSE" | sed '$d')
+            if [[ "$T_CODE" -eq 201 ]]; then
+              export GH_TOKEN=$(echo "$T_BODY" | python3 -c "import sys,json; print(json.load(sys.stdin)['token'])")
+              USE_APP_TOKEN="true"
+              echo "[auth] GitHub App token generated successfully"
+            else
+              echo "[auth] WARN: App token creation failed (HTTP ${T_CODE}), falling back to GITHUB_TOKEN"
+            fi
+          else
+            echo "[auth] WARN: App not installed on ${REPO_OWNER}/${REPO_NAME} (HTTP ${HTTP_CODE}), falling back to GITHUB_TOKEN"
+          fi
+        else
+          echo "[auth] GitHub App credentials not mounted, using GITHUB_TOKEN"
+        fi
+
+        if [[ "$USE_APP_TOKEN" != "true" ]]; then
+          if [[ -z "${GH_TOKEN:-}" && -z "${GITHUB_TOKEN:-}" ]]; then
+            echo "[auth] ERROR: No GitHub token available (App token failed and GITHUB_TOKEN not set)" >&2
+            exit 1
+          fi
+          export GH_TOKEN="${GH_TOKEN:-${GITHUB_TOKEN}}"
+          echo "[auth] Using GITHUB_TOKEN (Phase 1 report-only — sufficient for read + comment)"
+          echo "[auth] NOTE: Phase 2+ auto-fix pushes require the GitHub App to be installed on ${REPO_OWNER}/${REPO_NAME}"
+        fi
+
+        # --- GCP auth for Claude (Vertex AI) fallback ---
+        export GOOGLE_APPLICATION_CREDENTIALS="/var/run/gcloud-adc/application_default_credentials.json"
+        export CLAUDE_CODE_USE_VERTEX="1"
+        export CLOUD_ML_REGION="global"
+        export ANTHROPIC_VERTEX_PROJECT_ID="itpc-gcp-hcm-pe-eng-claude"
+
+        # --- Run CI monitor ---
+        export PR_URL="https://github.com/${REPO_OWNER}/${REPO_NAME}/pull/${PULL_NUMBER}"
+        export SKIP_POLL="false"
+        export SELF_JOB_NAME="oape-ci-monitor"
+        export BUILD_ID="${BUILD_ID:-}"
+        export OAPE_RUN_URL="${BUILD_LOG_URL:-}"
+
+        gh auth setup-git
+        /app/scripts/ci-monitor/monitor.sh
+        /app/scripts/ci-monitor/dispatch.sh
+      credentials:
+      - mount_path: /var/run/gcloud-adc
+        name: oap-lts-claude-gcp-vertex-sa
+        namespace: test-credentials
+      - mount_path: /var/run/github-app
+        name: openshift-app-platform-shift-github-bot
+        namespace: test-credentials
+      from: ci-monitor-agent
+      resources:
+        requests:
+          cpu: "1"
+          memory: 500Mi
+      timeout: 2h30m0s

go.work.sum

@@ -1,19 +1,44 @@
+cloud.google.com/go/compute/metadata v0.3.0 h1:Tz+eQXMEqDIKRsmY3cHTL6FVaynIjX2QxYC4trgAKZc=
 cloud.google.com/go/compute/metadata v0.3.0/go.mod h1:zFmK7XCadkQkj6TtorcaGlCW1hT1fIilQDwofLpJ20k=
+github.com/NYTimes/gziphandler v0.0.0-20170623195520-56545f4a5d46 h1:lsxEuwrXEAokXB9qhlbKWPpo3KMLZQ5WB5WLQRW1uq0=
 github.com/NYTimes/gziphandler v0.0.0-20170623195520-56545f4a5d46/go.mod h1:3wb06e3pkSAbeQ52E9H9iFoQsEEwGN64994WTCIhntQ=
+github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5 h1:0CwZNZbxp69SHPdPJAN/hZIm0C4OItdklCFmMRWYpio=
 github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5/go.mod h1:wHh0iHkYZB8zMSxRWpUBQtwG5a7fFgvEO+odwuTv2gs=
+github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a h1:idn718Q4B6AGu/h5Sxe66HYVdqdGu2l9Iebqhi/AEoA=
 github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a/go.mod h1:lB+ZfQJz7igIIfQNfa7Ml4HSf2uFQQRzpGGRXenZAgY=
+github.com/creack/pty v1.1.9 h1:uDmaGzcdjhF4i/plgjmEsriH11Y0o7RKapEf/LDaM3w=
+github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da h1:oI5xCqsCo564l8iNU+DwB5epxmsaqB+rhGL0m5jtYqE=
 github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
+github.com/google/btree v1.0.1 h1:gK4Kx5IaGY9CD5sPJ36FHiBJ6ZXl0kilRiiCj+jdYp4=
 github.com/google/btree v1.0.1/go.mod h1:xXMiIv4Fb/0kKde4SpL7qlzvu5cMJDRkFDxJfI9uaxA=
+github.com/gorilla/websocket v1.5.0 h1:PPwGk2jz7EePpoHN/+ClbZu8SPxiqlu12wZP/3sWmnc=
 github.com/gorilla/websocket v1.5.0/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
+github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7 h1:pdN6V1QBWetyv/0+wjACpqVH+eVULgEjkurDLq3goeM=
 github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7/go.mod h1:FecbI9+v66THATjSRHfNgh1IVFe/9kFxbXtjV0ctIMA=
+github.com/kisielk/errcheck v1.5.0 h1:e8esj/e4R+SAOwFwN+n3zr0nYeCyeweozKfO23MvHzY=
+github.com/kisielk/gotool v1.0.0 h1:AV2c/EiW3KqPNT9ZKl07ehoAGi4C5/01Cfbblndcapg=
+github.com/kr/pty v1.1.1 h1:VkoXIwSboBpnk99O/KFauAEILuNHv5DVFKZMBN/gUgw=
+github.com/moby/spdystream v0.4.0 h1:Vy79D6mHeJJjiPdFEL2yku1kl0chZpJfZcPpb16BRl8=
 github.com/moby/spdystream v0.4.0/go.mod h1:xBAYlnt/ay+11ShkdFKNAG7LsyK/tmNBVvVOwrfMgdI=
+github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f h1:y5//uYreIhSUg3J1GEMiLbxo1LJaP8RfCpH6pymGZus=
 github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f/go.mod h1:ZdcZmHo+o7JKHSa8/e818NopupXU1YMK5fe1lsApnBw=
+github.com/peterbourgon/diskv v2.0.1+incompatible h1:UBdAOUP5p4RWqPBg048CAvpKN+vxiaj6gdUUzhl4XmI=
 github.com/peterbourgon/diskv v2.0.1+incompatible/go.mod h1:uqqh8zWWbv1HBMNONnaR/tNboyR3/BZd58JJSHlUSCU=
+github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
 github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
+github.com/stretchr/objx v0.5.0 h1:1zr/of2m5FGMsad5YfcqgdqdWrIhu+EBEJRhR1U7z/c=
+github.com/yuin/goldmark v1.2.1 h1:ruQGxdhGHe7FWOJPT0mKs5+pD2Xs1Bm/kdGlHO04FmM=
+go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
 go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
+golang.org/x/crypto v0.24.0 h1:mnl8DM0o513X8fdIkmyFE/5hTYxbwYOjDS/+rK6qpRI=
 golang.org/x/crypto v0.24.0/go.mod h1:Z1PMYSOR5nyMcyAVAIQSKCDwalqy85Aqn1x3Ws4L5DM=
+golang.org/x/mod v0.17.0 h1:zY54UmvipHiNd+pm+m0x9KhZ9hl1/7QNMyxXbc6ICqA=
 golang.org/x/mod v0.17.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
+golang.org/x/sync v0.7.0 h1:YsImfSBoP9QPYL0xyKJPq0gcaJdG3rInoqxTWbfQu9M=
 golang.org/x/sync v0.7.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
+golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 h1:H2TDz8ibqkAF6YGhCdN3jS9O0/s90v0rJh3X/OLHEUk=
 golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2/go.mod h1:K8+ghG5WaK9qNqU5K3HdILfMLy1f3aNYFI/wnl100a8=
+gopkg.in/evanphx/json-patch.v4 v4.12.0 h1:n6jtcsulIzXPJaxegRbvFNNrZDjbij7ny3gmSPG+6V4=
 gopkg.in/evanphx/json-patch.v4 v4.12.0/go.mod h1:p8EYWUEYMpynmqDbY58zCKCFZw8pRWMG4EsWvDvM72M=
+k8s.io/gengo/v2 v2.0.0-20240228010128-51d4e06bde70 h1:NGrVE502P0s0/1hudf8zjgwki1X/TByhmAoILTarmzo=
 k8s.io/gengo/v2 v2.0.0-20240228010128-51d4e06bde70/go.mod h1:VH3AT8AaQOqiGjMF9p0/IM1Dj+82ZwjfxUP1IxaHE+8=