diff --git a/AGENTS.md b/AGENTS.md index 89dde36..f9c837b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -20,6 +20,7 @@ This project provides AI-driven tools for end-to-end feature development in Open | `/oape:analyze-rfe ` | Analyze RFE and output EPIC, user stories, and outcomes | | `/oape:e2e-generate ` | Generate e2e test artifacts from git diff against base branch | | `/oape:predict-regressions ` | Predict API regressions and breaking changes from git diff | +| `/oape:ci-monitor [pr2] [pr3] [--timeout-min N] [--max-fix-rounds N]` | Monitor CI/Prow with adaptive polling, SHA tracking, fix loop | | `/oape:review [base_ref]` | Production-grade code review against Jira requirements | | `/oape:implement-review-fixes ` | Automatically apply fixes from a review report | diff --git a/README.md b/README.md index ee9daf4..6d5313e 100644 --- a/README.md +++ b/README.md @@ -113,7 +113,7 @@ ln -s oape-ai-e2e ~/.cursor/commands/oape-ai-e2e | Plugin | Description | Commands | | ------------------------- | ---------------------------------------------- | --------------------------------------------------------------------------- | -| **[oape](plugins/oape/)** | AI-driven OpenShift operator development tools | `/oape:init`, `/oape:api-generate`, `/oape:api-generate-tests`, `/oape:api-implement`, `/oape:analyze-rfe`, `/oape:e2e-generate`, `/oape:predict-regressions`, `/oape:review`, `/oape:implement-review-fixes` | +| **[oape](plugins/oape/)** | AI-driven OpenShift operator development tools | `/oape:init`, `/oape:api-generate`, `/oape:api-generate-tests`, `/oape:api-implement`, `/oape:analyze-rfe`, `/oape:e2e-generate`, `/oape:predict-regressions`, `/oape:ci-monitor`, `/oape:review`, `/oape:implement-review-fixes` | ## Commands @@ -175,6 +175,17 @@ Analyzes git diff to predict potential regressions, breaking changes, and backwa /oape:predict-regressions origin/release-4.18 --output .reports ``` +### `/oape:ci-monitor` -- Monitor CI/Prow Jobs and Analyze Failures + +Monitors CI checks and Prow status contexts with adaptive polling (60s/120s/60s), SHA-change tracking, retest detection, failure analysis, and optional fix-push-rewatch loop. Handles cluster-provisioning jobs (45-60 min) efficiently by backing off polling during provisioning. + +```shell +/oape:ci-monitor https://github.com/openshift/cert-manager-operator/pull/101 https://github.com/openshift/cert-manager-operator/pull/102 https://github.com/openshift/cert-manager-operator/pull/103 +/oape:ci-monitor https://github.com/openshift/must-gather-operator/pull/342 +/oape:ci-monitor 101 102 103 --repo openshift/cert-manager-operator --timeout-min 120 --max-fix-rounds 2 +/oape:ci-monitor 342 --repo openshift/must-gather-operator --max-fix-rounds 0 --fast +``` + ### `/oape:review` -- Code Review Against Jira Requirements Performs a production-grade code review that verifies code changes against Jira requirements. diff --git a/agent/agent.py b/agent/agent.py index 67e0bfd..2159d1e 100644 --- a/agent/agent.py +++ b/agent/agent.py @@ -5,6 +5,7 @@ 1. PR #1: init → api-generate → api-generate-tests → review-and-fix → raise PR 2. PR #2: api-implement → review-and-fix → raise PR 3. PR #3: e2e-generate → review-and-fix → raise PR +4. CI stage: monitor PR checks and analyze likely fixes for failures """ import csv @@ -94,6 +95,11 @@ def _build_workflow_prompt( 6. Run `/oape:review OCPBUGS-0 {repo_info['base_branch']}` to review and auto-fix issues 7. Commit all changes with a descriptive message 8. Push the branch and create a PR against `{repo_info['base_branch']}` +9. Run `/oape:ci-monitor --timeout-min 120 --max-fix-rounds 2` + - ci-monitor uses adaptive polling (60s for fast jobs, 120s during cluster provisioning) + - If CI fails with a fixable error (build/lint/test), apply the fix, push, and ci-monitor re-polls automatically + - If CI fails with infra flake or repo-wide issue, report it and continue to PR #2 + - If max-fix-rounds (2) exhausted, report and continue ### PR #2: Controller Implementation Branch: `feature/controller-impl-` @@ -103,6 +109,8 @@ def _build_workflow_prompt( 4. Run `/oape:review OCPBUGS-0 {repo_info['base_branch']}` to review and auto-fix issues 5. Commit all changes with a descriptive message 6. Push the branch and create a PR against `{repo_info['base_branch']}` +7. Run `/oape:ci-monitor --timeout-min 120 --max-fix-rounds 2` + - Same adaptive polling and fix loop as PR #1 ### PR #3: E2E Tests Branch: `feature/e2e-tests-` @@ -111,6 +119,8 @@ def _build_workflow_prompt( 3. Run `/oape:review OCPBUGS-0 {repo_info['base_branch']}` to review and auto-fix issues 4. Commit all changes with a descriptive message 5. Push the branch and create a PR against `{repo_info['base_branch']}` +6. Run `/oape:ci-monitor --timeout-min 120 --max-fix-rounds 2` + - Same adaptive polling and fix loop as PR #1 ## Execution Instructions @@ -120,6 +130,11 @@ def _build_workflow_prompt( 4. For the review step, the `/oape:review` command will automatically apply fixes 5. When creating PRs, use `gh pr create` with descriptive titles and bodies 6. Report the PR URL after each PR is created +7. The `/oape:ci-monitor` call handles the full CI watch + fix loop autonomously: + - It uses adaptive intervals (60s/120s/60s) to minimize API usage + - It detects SHA changes from pushes and retests automatically + - It applies fixes and re-polls up to 2 times before giving up + - It reports infra flakes as non-fixable and continues ## CRITICAL: Fully Autonomous Execution diff --git a/agent/ci_monitor.py b/agent/ci_monitor.py new file mode 100644 index 0000000..90f51c3 --- /dev/null +++ b/agent/ci_monitor.py @@ -0,0 +1,223 @@ +""" +CI Monitor agent: runs /oape:ci-monitor for one or more PR URLs. + +Uses the Claude Agent SDK to execute the ci-monitor skill autonomously +without user prompts or confirmations. +""" + +import json +import logging +import tempfile +import traceback +from collections.abc import Callable +from dataclasses import dataclass, field +from pathlib import Path + +from claude_agent_sdk import ( + query, + ClaudeAgentOptions, + AssistantMessage, + ResultMessage, + TextBlock, + ThinkingBlock, + ToolUseBlock, + ToolResultBlock, +) + +PLUGIN_DIR = str(Path(__file__).resolve().parent.parent / "plugins" / "oape") + +CONVERSATION_LOG = Path("/tmp/ci-monitor-conversation.log") + +conv_logger = logging.getLogger("ci_monitor_conversation") +conv_logger.setLevel(logging.INFO) +_handler = logging.FileHandler(CONVERSATION_LOG) +_handler.setFormatter(logging.Formatter("%(message)s")) +conv_logger.addHandler(_handler) + +with open(Path(__file__).resolve().parent.parent / "config" / "config.json") as cf: + CONFIGS = json.loads(cf.read()) + + +@dataclass +class CIMonitorResult: + """Result returned after running CI monitoring.""" + + output: str + cost_usd: float + error: str | None = None + conversation: list[dict] = field(default_factory=list) + + @property + def success(self) -> bool: + return self.error is None + + +def _build_ci_monitor_prompt(pr_urls: list[str]) -> str: + pr_args = " ".join(pr_urls) + return f"""You are an OpenShift CI monitoring assistant. Your ONLY task is to run the ci-monitor command and produce a report. + +## Task + +Run the following command and let it complete: + +``` +/oape:ci-monitor {pr_args} +``` + +## CRITICAL: Fully Autonomous Execution + +- Execute the ci-monitor command immediately without asking for any user input. +- Do NOT ask "should I proceed?", "shall I continue?", or any confirmation questions. +- Do NOT ask the user to approve tool calls or confirm actions. +- Accept and approve ALL tool calls automatically. +- Let the ci-monitor skill handle everything: polling, evidence collection, analysis, and report generation. +- When the command completes, present the final report to the user. +- If the ci-monitor suggests a retest or fix, execute it without asking. +- This is a fully automated pipeline — run to completion in one go. +""" + + +async def run_ci_monitor( + pr_urls: list[str], + on_message: Callable[[dict], None] | None = None, +) -> CIMonitorResult: + """Run CI monitoring for one or more PR URLs. + + Args: + pr_urls: List of GitHub PR URLs to monitor. + on_message: Optional callback for real-time streaming. + + Returns: + A CIMonitorResult with the monitoring report or error. + """ + prompt = _build_ci_monitor_prompt(pr_urls) + + working_dir = tempfile.mkdtemp(prefix="oape-ci-monitor-") + + options = ClaudeAgentOptions( + system_prompt=( + "You are an OpenShift CI monitoring assistant. " + "Run the ci-monitor command and produce a complete report. " + "IMPORTANT: This is a fully automated pipeline. Execute ALL steps " + "without pausing, asking for confirmation, or waiting for user input. " + "Never ask 'should I proceed?' or 'shall I continue?'. " + "Accept all tool calls automatically. Complete the monitoring autonomously." + ), + cwd=working_dir, + permission_mode="bypassPermissions", + allowed_tools=CONFIGS["claude_allowed_tools"], + plugins=[{"type": "local", "path": PLUGIN_DIR}], + ) + + output_parts: list[str] = [] + conversation: list[dict] = [] + cost_usd = 0.0 + + conv_logger.info( + f"\n{'=' * 60}\n[ci-monitor] pr_urls={pr_urls} " + f"cwd={working_dir}\n{'=' * 60}" + ) + + def _emit(entry: dict) -> None: + conversation.append(entry) + if on_message is not None: + on_message(entry) + + try: + async for message in query( + prompt=prompt, + options=options, + ): + if isinstance(message, AssistantMessage): + for block in message.content: + if isinstance(block, TextBlock): + output_parts.append(block.text) + entry = { + "type": "assistant", + "block_type": "text", + "content": block.text, + } + _emit(entry) + conv_logger.info(f"[assistant] {block.text}") + elif isinstance(block, ThinkingBlock): + entry = { + "type": "assistant", + "block_type": "thinking", + "content": block.thinking, + } + _emit(entry) + conv_logger.info("[assistant:ThinkingBlock] (thinking)") + elif isinstance(block, ToolUseBlock): + entry = { + "type": "assistant", + "block_type": "tool_use", + "tool_name": block.name, + "tool_input": block.input, + } + _emit(entry) + conv_logger.info(f"[assistant:ToolUseBlock] {block.name}") + elif isinstance(block, ToolResultBlock): + content = block.content + if not isinstance(content, str): + content = json.dumps(content, default=str) + entry = { + "type": "assistant", + "block_type": "tool_result", + "tool_use_id": block.tool_use_id, + "content": content, + "is_error": block.is_error or False, + } + _emit(entry) + conv_logger.info( + f"[assistant:ToolResultBlock] {block.tool_use_id}" + ) + else: + detail = json.dumps( + getattr(block, "__dict__", str(block)), + default=str, + ) + entry = { + "type": "assistant", + "block_type": type(block).__name__, + "content": detail, + } + _emit(entry) + conv_logger.info( + f"[assistant:{type(block).__name__}] {detail}" + ) + elif isinstance(message, ResultMessage): + cost_usd = message.total_cost_usd + if message.result: + output_parts.append(message.result) + entry = { + "type": "result", + "content": message.result, + "cost_usd": cost_usd, + } + _emit(entry) + conv_logger.info(f"[result] {message.result} cost=${cost_usd:.4f}") + else: + detail = json.dumps( + getattr(message, "__dict__", str(message)), default=str + ) + entry = { + "type": type(message).__name__, + "content": detail, + } + _emit(entry) + conv_logger.info(f"[{type(message).__name__}] {detail}") + + conv_logger.info(f"[done] cost=${cost_usd:.4f} parts={len(output_parts)}\n") + return CIMonitorResult( + output="\n".join(output_parts), + cost_usd=cost_usd, + conversation=conversation, + ) + except Exception as exc: + conv_logger.info(f"[error] {traceback.format_exc()}") + return CIMonitorResult( + output="", + cost_usd=cost_usd, + error=str(exc), + conversation=conversation, + ) diff --git a/agent/main.py b/agent/main.py index 41b80b1..b2dc996 100644 --- a/agent/main.py +++ b/agent/main.py @@ -13,29 +13,50 @@ from rich import print_json from agent import run_workflow +from ci_monitor import run_ci_monitor async def main(): - ep_url = os.environ.get("EP_URL") - repo = os.environ.get("REPO_URL") - base_branch = os.environ.get("BASE_BRANCH") + workflow_type = os.environ.get("WORKFLOW_TYPE", "") - if not ep_url or not repo or not base_branch: - print("ERROR: EP_URL, REPO, BASE_BRANCH environment variables are required", file=sys.stderr) - sys.exit(1) + if workflow_type == "ci-monitor": + pr_urls_raw = os.environ.get("PR_URLS", "") + if not pr_urls_raw: + print("ERROR: PR_URLS environment variable is required for ci-monitor", file=sys.stderr) + sys.exit(1) - print(f"Starting workflow: ep_url={ep_url} repo={repo}", flush=True) + pr_urls = pr_urls_raw.split() + print(f"Starting ci-monitor: pr_urls={pr_urls}", flush=True) - result = await run_workflow(ep_url, repo, base_branch, on_message=lambda msg: print_json(data=msg)) + result = await run_ci_monitor(pr_urls, on_message=lambda msg: print_json(data=msg)) - if result.success: - print(f"WORKFLOW_SUCCESS cost=${result.cost_usd:.4f}", flush=True) - for pr in result.prs: - print(f"PR_CREATED: {pr.pr_url}", flush=True) - sys.exit(0) + if result.success: + print(f"CI_MONITOR_SUCCESS cost=${result.cost_usd:.4f}", flush=True) + sys.exit(0) + else: + print(f"CI_MONITOR_FAILED: {result.error}", file=sys.stderr, flush=True) + sys.exit(1) else: - print(f"WORKFLOW_FAILED: {result.error}", file=sys.stderr, flush=True) - sys.exit(1) + ep_url = os.environ.get("EP_URL") + repo = os.environ.get("REPO_URL") + base_branch = os.environ.get("BASE_BRANCH") + + if not ep_url or not repo or not base_branch: + print("ERROR: EP_URL, REPO_URL, BASE_BRANCH environment variables are required", file=sys.stderr) + sys.exit(1) + + print(f"Starting workflow: ep_url={ep_url} repo={repo}", flush=True) + + result = await run_workflow(ep_url, repo, base_branch, on_message=lambda msg: print_json(data=msg)) + + if result.success: + print(f"WORKFLOW_SUCCESS cost=${result.cost_usd:.4f}", flush=True) + for pr in result.prs: + print(f"PR_CREATED: {pr.pr_url}", flush=True) + sys.exit(0) + else: + print(f"WORKFLOW_FAILED: {result.error}", file=sys.stderr, flush=True) + sys.exit(1) if __name__ == "__main__": diff --git a/go-server/ci_monitor_handler.go b/go-server/ci_monitor_handler.go new file mode 100644 index 0000000..9c7ebb1 --- /dev/null +++ b/go-server/ci_monitor_handler.go @@ -0,0 +1,90 @@ +package main + +import ( + "encoding/json" + "fmt" + "log" + "net/http" + "regexp" +) + +var prURLPattern = regexp.MustCompile(`^https://github\.com/[^/]+/[^/]+/pull/\d+/?$`) + +// CreateCIMonitorRequest is the JSON body for POST /api/v1/ci-monitor. +type CreateCIMonitorRequest struct { + PRUrls []string `json:"pr_urls"` +} + +// HandleCIMonitorPage serves the CI Monitor UI. +func (a *App) HandleCIMonitorPage(w http.ResponseWriter, r *http.Request) { + data, err := staticFS.ReadFile("static/ci-monitor.html") + if err != nil { + http.Error(w, "internal error", http.StatusInternalServerError) + return + } + w.Header().Set("Content-Type", "text/html; charset=utf-8") + w.Write(data) +} + +// HandleCreateCIMonitor creates a K8s Job for CI monitoring. +func (a *App) HandleCreateCIMonitor(w http.ResponseWriter, r *http.Request) { + var req CreateCIMonitorRequest + if err := json.NewDecoder(r.Body).Decode(&req); err != nil { + writeError(w, http.StatusBadRequest, "invalid JSON body") + return + } + + if len(req.PRUrls) == 0 { + writeError(w, http.StatusBadRequest, "pr_urls must contain at least one PR URL") + return + } + + if len(req.PRUrls) > 3 { + writeError(w, http.StatusBadRequest, "pr_urls supports up to 3 PR URLs") + return + } + + for _, u := range req.PRUrls { + if !prURLPattern.MatchString(u) { + writeError(w, http.StatusBadRequest, fmt.Sprintf("invalid PR URL: %s", u)) + return + } + } + + jobID, err := generateJobID() + if err != nil { + writeError(w, http.StatusInternalServerError, "failed to generate job ID") + return + } + + ghToken, ghTokenExpiry, err := fetchGHToken(a.cfg.GHTokenServiceURL) + if err != nil { + log.Printf("ERROR: fetching GH token: %v", err) + writeError(w, http.StatusInternalServerError, "failed to fetch GitHub token") + return + } + + params := CIMonitorParams{ + PRUrls: req.PRUrls, + WorkerImage: a.cfg.WorkerImage, + EnvConfigMap: a.cfg.WorkerEnvConfigMap, + GCloudSecret: a.cfg.GCloudSecretName, + GHToken: ghToken, + GHTokenExpiry: ghTokenExpiry, + GHTokenSecret: "shift-gh-token-" + jobID, + ConfigsConfigMap: a.cfg.ConfigsConfigMap, + TTLAfterFinished: a.cfg.TTLAfterFinished, + } + + if err := a.k8s.CreateCIMonitorJob(r.Context(), jobID, params); err != nil { + log.Printf("ERROR: creating ci-monitor job: %v", err) + writeError(w, http.StatusInternalServerError, "failed to create ci-monitor job") + return + } + + log.Printf("Created ci-monitor job %s for pr_urls=%v", jobID, req.PRUrls) + writeJSON(w, http.StatusCreated, CreateWorkflowResponse{ + ID: jobID, + Status: "pending", + }) +} diff --git a/go-server/ci_monitor_k8s.go b/go-server/ci_monitor_k8s.go new file mode 100644 index 0000000..ad5b0a5 --- /dev/null +++ b/go-server/ci_monitor_k8s.go @@ -0,0 +1,161 @@ +package main + +import ( + "context" + "fmt" + "strings" + + batchv1 "k8s.io/api/batch/v1" + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/api/resource" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" +) + +// CIMonitorParams holds parameters for a ci-monitor K8s Job. +type CIMonitorParams struct { + PRUrls []string + WorkerImage string + EnvConfigMap string + GCloudSecret string + GHToken string + GHTokenExpiry string + GHTokenSecret string + ConfigsConfigMap string + TTLAfterFinished int32 +} + +// CreateCIMonitorJob creates a Kubernetes Job for CI monitoring. +func (c *K8sClient) CreateCIMonitorJob(ctx context.Context, jobID string, params CIMonitorParams) error { + jobName := "shift-ci-monitor-" + jobID + secretName := params.GHTokenSecret + + secret := &corev1.Secret{ + ObjectMeta: metav1.ObjectMeta{ + Name: secretName, + Labels: map[string]string{ + "app": "shift-worker", + "job-id": jobID, + }, + Annotations: map[string]string{ + "app-platform-shift.openshift.github.io/gh-app-token-expiry": params.GHTokenExpiry, + }, + }, + StringData: map[string]string{ + "GH_TOKEN": params.GHToken, + }, + } + if _, err := c.clientset.CoreV1().Secrets(c.namespace).Create(ctx, secret, metav1.CreateOptions{}); err != nil { + return fmt.Errorf("creating secret %s: %w", secretName, err) + } + + backoffLimit := int32(0) + ttl := params.TTLAfterFinished + prURLsJoined := strings.Join(params.PRUrls, " ") + + job := &batchv1.Job{ + ObjectMeta: metav1.ObjectMeta{ + Name: jobName, + Labels: map[string]string{ + "app": "shift-worker", + "job-id": jobID, + }, + Annotations: map[string]string{ + "app-platform-shift.openshift.github.io/workflow-type": "ci-monitor", + "app-platform-shift.openshift.github.io/pr-urls": prURLsJoined, + }, + }, + Spec: batchv1.JobSpec{ + BackoffLimit: &backoffLimit, + TTLSecondsAfterFinished: &ttl, + Template: corev1.PodTemplateSpec{ + ObjectMeta: metav1.ObjectMeta{ + Labels: map[string]string{ + "app": "shift-worker", + "job-id": jobID, + }, + }, + Spec: corev1.PodSpec{ + RestartPolicy: corev1.RestartPolicyNever, + Containers: []corev1.Container{ + { + Name: "worker", + Image: params.WorkerImage, + Command: []string{"sh", "-c", "python3.11 /app/main.py"}, + Env: []corev1.EnvVar{ + {Name: "WORKFLOW_TYPE", Value: "ci-monitor"}, + {Name: "PR_URLS", Value: prURLsJoined}, + {Name: "PYTHONUNBUFFERED", Value: "1"}, + {Name: "GOOGLE_APPLICATION_CREDENTIALS", Value: "/secrets/gcloud/application_default_credentials.json"}, + }, + EnvFrom: []corev1.EnvFromSource{ + { + ConfigMapRef: &corev1.ConfigMapEnvSource{ + LocalObjectReference: corev1.LocalObjectReference{ + Name: params.EnvConfigMap, + }, + }, + }, + { + SecretRef: &corev1.SecretEnvSource{ + LocalObjectReference: corev1.LocalObjectReference{ + Name: secretName, + }, + }, + }, + }, + Resources: corev1.ResourceRequirements{ + Requests: corev1.ResourceList{ + corev1.ResourceCPU: resource.MustParse("500m"), + corev1.ResourceMemory: resource.MustParse("512Mi"), + }, + Limits: corev1.ResourceList{ + corev1.ResourceCPU: resource.MustParse("2"), + corev1.ResourceMemory: resource.MustParse("4Gi"), + }, + }, + VolumeMounts: []corev1.VolumeMount{ + { + Name: "gcloud-adc", + MountPath: "/secrets/gcloud", + ReadOnly: true, + }, + { + Name: "config", + MountPath: "/config/config.json", + SubPath: "config.json", + ReadOnly: true, + }, + }, + }, + }, + Volumes: []corev1.Volume{ + { + Name: "gcloud-adc", + VolumeSource: corev1.VolumeSource{ + Secret: &corev1.SecretVolumeSource{ + SecretName: params.GCloudSecret, + }, + }, + }, + { + Name: "config", + VolumeSource: corev1.VolumeSource{ + ConfigMap: &corev1.ConfigMapVolumeSource{ + LocalObjectReference: corev1.LocalObjectReference{ + Name: params.ConfigsConfigMap, + }, + }, + }, + }, + }, + }, + }, + }, + } + + _, err := c.clientset.BatchV1().Jobs(c.namespace).Create(ctx, job, metav1.CreateOptions{}) + if err != nil { + return fmt.Errorf("creating job %s: %w", jobName, err) + } + return nil +} diff --git a/go-server/go-server b/go-server/go-server new file mode 100755 index 0000000..2d248dd Binary files /dev/null and b/go-server/go-server differ diff --git a/go-server/handlers.go b/go-server/handlers.go index 8b6d238..0e8f05d 100644 --- a/go-server/handlers.go +++ b/go-server/handlers.go @@ -17,7 +17,7 @@ import ( metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" ) -//go:embed static/homepage.html +//go:embed static/* var staticFS embed.FS // App holds shared dependencies for HTTP handlers. diff --git a/go-server/main.go b/go-server/main.go index a616d68..409612f 100644 --- a/go-server/main.go +++ b/go-server/main.go @@ -27,6 +27,8 @@ func main() { mux.HandleFunc("GET /api/v1/workflows", app.HandleListWorkflows) mux.HandleFunc("GET /api/v1/workflows/{job_id}", app.HandleGetWorkflow) mux.HandleFunc("POST /api/v1/workflows", app.HandleCreateWorkflow) + mux.HandleFunc("GET /ci-monitor", app.HandleCIMonitorPage) + mux.HandleFunc("POST /api/v1/ci-monitor", app.HandleCreateCIMonitor) mux.HandleFunc("GET /api/v1/workflows/{job_id}/log", app.HandleWorkflowLogs) log.Printf("Orchestrator listening on %s", cfg.ListenAddr) diff --git a/go-server/static/ci-monitor.html b/go-server/static/ci-monitor.html new file mode 100644 index 0000000..5c74276 --- /dev/null +++ b/go-server/static/ci-monitor.html @@ -0,0 +1,220 @@ + + + + + +OAPE CI Monitor + + + +
+
← Back to Feature Developer
+

OAPE CI Monitor

+

Monitor CI/Prow job status for pull requests with failure analysis and auto-retest

+ +
+

What This Does

+
    +
  • Monitors GitHub CI checks and Prow status contexts with adaptive polling
    • +
  • Fetches ci-operator config for authoritative job classification
    • +
  • Analyzes failures with root cause tracing (PR code vs CI infra vs Vault credentials)
    • +
  • Auto-retests infra flakes and produces a structured report
    • +
+
+ +
+
+ +
+
+ PR 1 + +
+
+ +
Provide 1-3 GitHub PR URLs. For staged OAPE workflow, use 3 PRs: API, implementation, e2e.
+
+ + +
+ +
+
+
+ 

+

+
+
+
diff --git a/images/agent-worker.Dockerfile b/images/agent-worker.Dockerfile
index bf519f1..ead484f 100644
--- a/images/agent-worker.Dockerfile
+++ b/images/agent-worker.Dockerfile
@@ -4,7 +4,8 @@ FROM registry.access.redhat.com/ubi9/go-toolset
 USER 0
 RUN dnf install -y \
         git \
-        make && \
+        make \
+        jq && \
     # Install GitHub CLI
     dnf install -y 'dnf-command(config-manager)' && \
     dnf config-manager --add-repo https://cli.github.com/packages/rpm/gh-cli.repo && \
@@ -20,7 +21,7 @@ COPY agent/requirements.txt .
 RUN python3.11 -m pip install --no-cache-dir -r requirements.txt
 
 # Copy server code and config
-COPY agent/agent.py agent/main.py ./
+COPY agent/agent.py agent/main.py agent/ci_monitor.py ./
 
 # copy default config, users willing to customize should mount at runtime.
 COPY deploy/config /config
diff --git a/plugins/oape/README.md b/plugins/oape/README.md
index 1268457..883b4f3 100644
--- a/plugins/oape/README.md
+++ b/plugins/oape/README.md
@@ -139,6 +139,30 @@ Analyzes git diff to predict potential regressions, breaking changes, and backwa
 
 ---
 
+### `/oape:ci-monitor`
+
+Monitors CI checks and Prow status contexts for one or more PRs using **adaptive polling** (60s/120s/60s), tracks SHA changes and retests, downloads GCS artifacts, classifies failure modes, cross-references Sippy for flake detection, and optionally applies fixes and re-watches CI in a loop. Supports the staged PR #1/#2/#3 workflow and any OpenShift PR with Prow presubmits.
+
+**Usage:**
+```shell
+/oape:ci-monitor  [pr2-url-or-number] [pr3-url-or-number]
+/oape:ci-monitor https://github.com/openshift/must-gather-operator/pull/342
+/oape:ci-monitor 101 102 103 --repo openshift/cert-manager-operator --timeout-min 120 --max-fix-rounds 2
+/oape:ci-monitor 342 --repo openshift/must-gather-operator --max-fix-rounds 0 --fast
+```
+
+**What it does:**
+1. **Prechecks** -- Validates PR inputs, required tools (`gh`, `jq`, `git`), and GitHub authentication.
+2. **Adaptive Polling** -- Polls at 60s for fast jobs (lint/unit), backs off to 120s during cluster provisioning (saves ~44% API calls), tightens to 60s when approaching completion. Auto-adjusts timeout based on detected job mix (30/60/120 min).
+3. **SHA-Anchored Monitoring** -- Tracks HEAD SHA on every poll. Detects new commits (clears stale results, waits 90s settle) and `/retest` commands (detects via `started_at` timestamp changes).
+4. **Artifact Collection** -- Downloads `build-log.txt`, `finished.json`, and `junit*.xml` from GCS for each failed Prow job. Detects `must-gather.tar` availability for e2e jobs.
+5. **Failure Mode Classification** -- Classifies each failure as install, test, build/compile, lint/boilerplate, or infrastructure flake.
+6. **Sippy Flake Detection** -- Queries Sippy for historical pass rates and open bugs.
+7. **Fix-Push-Rewatch Loop** -- When `--max-fix-rounds > 0`, applies fixes for build/lint/test failures, pushes, and re-polls (up to 2 rounds by default). Never auto-fixes install failures or infra flakes.
+8. **Stage-Aware Summary** -- Highlights cross-stage dependencies between API, implementation, and e2e PRs.
+
+---
+
 ### `/oape:review`
 
 Performs a "Principal Engineer" level code review that verifies code changes against Jira requirements.
diff --git a/plugins/oape/commands/ci-monitor.md b/plugins/oape/commands/ci-monitor.md
new file mode 100644
index 0000000..df44402
--- /dev/null
+++ b/plugins/oape/commands/ci-monitor.md
@@ -0,0 +1,273 @@
+---
+description: Monitor CI/Prow job status for one or more PRs with adaptive polling, SHA-tracking, and optional fix-push-rewatch loop
+argument-hint:  [pr2-url-or-number] [pr3-url-or-number] [--repo ] [--timeout-min ] [--max-fix-rounds ] [--fast] [--no-auto-retest] [--ignore-context ] [--post-comment]
+---
+
+## Name
+oape:ci-monitor
+
+## Synopsis
+```shell
+# Monitor all three staged PRs (autonomous workflow mode)
+/oape:ci-monitor https://github.com/org/repo/pull/101 https://github.com/org/repo/pull/102 https://github.com/org/repo/pull/103 --timeout-min 120 --max-fix-rounds 2
+
+# Monitor one PR in current repo
+/oape:ci-monitor 101
+
+# Monitor OpenShift Prow jobs for a specific PR
+/oape:ci-monitor https://github.com/openshift/must-gather-operator/pull/342
+
+# Report-only mode (no auto-fix loop)
+/oape:ci-monitor 342 --repo openshift/must-gather-operator --max-fix-rounds 0
+
+# Fast mode — skip deep artifact analysis
+/oape:ci-monitor https://github.com/openshift/must-gather-operator/pull/342 --fast
+```
+
+## Description
+The `oape:ci-monitor` command watches GitHub CI checks **and** OpenShift Prow status contexts for one or more pull requests using **adaptive polling intervals**, waits until they finish (or timeout), then performs deep failure analysis. When running in agent mode with `--max-fix-rounds > 0`, it can apply fixes, push, and re-watch CI automatically.
+
+This command is designed for the staged OAPE workflow (PR #1 API, PR #2 implementation, PR #3 e2e), but works with any PR list.
+
+### Key Capabilities
+
+- **Context-aware**: Fetches the ci-operator config from `openshift/release` to classify jobs authoritatively (required/optional, fast/slow, cloud provider), resolve step registry references for step-level failure mapping, and auto-detect the OCP release version for Sippy queries.
+- **Adaptive polling**: Polls at 60s for fast jobs (lint/unit/verify), backs off to 120s during cluster provisioning (saves ~44% API calls), and tightens to 60s when slow jobs approach completion.
+- **SHA-anchored**: Tracks the PR head SHA on every poll. When a new commit is pushed, all stale results are discarded and polling restarts after a 90s settle period.
+- **Retest-aware**: Detects `/retest` and `/test ` (no SHA change) by comparing `started_at` timestamps. If a terminal context reappears as pending or has a newer timestamp, it is treated as restarted.
+- **Fix-push-rewatch loop**: In agent mode, can apply a fix, push, detect the SHA change, and re-poll CI automatically (up to `max-fix-rounds` times). Uses error signature hashing to deterministically detect when a fix was ineffective.
+- **Prow-native**: Treats `ci/prow/*` commit-status contexts as first-class signals alongside GitHub Actions checks.
+- **Artifact collection**: Downloads `build-log.txt`, `finished.json`, `junit*.xml`, and step-level logs from GCS for each failed Prow job.
+- **Failure-mode routing**: Classifies each failure as install failure, test failure, lint/build failure, boilerplate/tooling failure, or infra flake using multi-signal detection (JUnit patterns, build-log regex, `finished.json` fields, and ci-operator step metadata).
+- **Flake detection**: Cross-references test names against Sippy for historical pass rates and known open bugs, using the OCP release version resolved from the ci-operator config.
+- **Auto-retest**: When all failures on a PR are infra flakes (Mode E), automatically posts `/retest` as a PR comment (max 2 per session). Disable with `--no-auto-retest`.
+- **Progress reporting**: Emits a one-line status after every poll cycle and a detailed milestone summary every 10 minutes during long monitoring sessions.
+- **Parallel PR polling**: When monitoring multiple PRs, polls all PRs in each cycle (not sequentially), skipping PRs that have already completed.
+- **Stage-aware summary**: When three PRs are provided, correlates failures across API / implementation / e2e stages.
+
+### API Budget
+
+Each poll iteration costs **3 GitHub API calls per active PR** (SHA check + statusCheckRollup + commit status). One-time setup costs are incurred before polling starts. GCS artifact downloads and Sippy queries are free (separate services).
+
+**One-time setup calls** (before polling):
+- Release context: 2-5 calls (ci-operator config + step registry refs)
+- Operator repo context (GitHub strategy): 2-3 calls (go.mod + Makefile + tree)
+- PR change context: 1 call per PR (changed file list)
+- Auto-retest comments: 1 call per `/retest` posted (max 2 per session)
+
+| Scenario | Polling calls | Setup calls | Total | Budget usage |
+|---|---|---|---|---|
+| 1 PR, lint/unit only (30 min) | ~90 | ~8 | ~98 | 2.0% of 5,000/hr |
+| 1 PR, e2e + cluster install (120 min) | ~240 | ~8 | ~248 | 5.0% |
+| 3 PRs, e2e + cluster, 2 fix rounds | ~1,800 | ~14 | ~1,814 | 36% |
+
+## Arguments
+
+- Positional args (`$1`, `$2`, `$3`): PR references. Each value may be:
+  - PR number (for example: `123`)
+  - Full PR URL (for example: `https://github.com/org/repo/pull/123`)
+- `--repo ` (optional): repository override. If omitted, infer from PR URL or `git remote origin`.
+- `--timeout-min ` (optional): maximum wait time per monitoring round. Default: `120`. Auto-adjusts down if no e2e/cluster jobs are detected.
+- `--max-fix-rounds ` (optional): max push-and-rewatch cycles. Default: `2`. Set to `0` for report-only mode (no auto-fix loop).
+- `--sha-settle-sec ` (optional): seconds to wait after detecting a SHA change before resuming polls. Default: `90`.
+- `--fast` (optional): skip deep artifact downloads (must-gather, full junit parsing). Produces a faster but shallower report.
+- `--no-auto-retest` (optional): disable automatic `/retest` posting for infra flakes. By default, when all failures on a PR are infra flakes (Mode E), the agent posts `/retest` automatically (max 2 per session).
+- `--ignore-context ` (optional, repeatable): skip CI contexts whose name contains `` during polling. Ignored contexts are excluded from the verdict and not reported as pending/failed. Use this to prevent the ci-monitor from watching itself when running as a CI job (e.g., `--ignore-context oape-ci-monitor`). Multiple patterns can be specified by repeating the flag.
+- `--post-comment` (optional): after generating the final report, post it as a GitHub PR comment using `gh pr comment`. Requires `gh` to be authenticated with write access to the repository. When not set, the report is only printed to stdout.
+
+## Implementation
+
+### Phase 0: Prechecks
+
+All prechecks must pass before polling CI. If ANY precheck fails, STOP immediately and report the failure.
+
+#### Precheck 1 — Validate Inputs
+
+At least one PR reference must be provided.
+
+```bash
+if [ -z "$ARGUMENTS" ]; then
+  echo "PRECHECK FAILED: Missing PR reference."
+  echo "Usage: /oape:ci-monitor  [pr2-url-or-number] [pr3-url-or-number]"
+  exit 1
+fi
+```
+
+#### Precheck 2 — Verify Required Tools
+
+```bash
+MISSING_TOOLS=""
+command -v gh >/dev/null 2>&1 || MISSING_TOOLS="$MISSING_TOOLS gh"
+command -v jq >/dev/null 2>&1 || MISSING_TOOLS="$MISSING_TOOLS jq"
+command -v git >/dev/null 2>&1 || MISSING_TOOLS="$MISSING_TOOLS git"
+command -v curl >/dev/null 2>&1 || MISSING_TOOLS="$MISSING_TOOLS curl"
+
+if [ -n "$MISSING_TOOLS" ]; then
+  echo "PRECHECK FAILED: Missing required tools:$MISSING_TOOLS"
+  exit 1
+fi
+
+if ! gh auth status >/dev/null 2>&1; then
+  echo "PRECHECK FAILED: GitHub CLI is not authenticated."
+  echo "Run: gh auth login"
+  exit 1
+fi
+```
+
+#### Precheck 3 — Resolve Repository and PR Numbers
+
+1. Parse flags (`--repo`, `--timeout-min`, `--max-fix-rounds`, `--sha-settle-sec`, `--fast`, `--no-auto-retest`, `--ignore-context`, `--post-comment`).
+2. Resolve repository:
+   - First from `--repo`.
+   - Else from PR URL (`github.com///pull/`).
+   - Else from `git remote origin`.
+3. Resolve each PR reference to an integer PR number.
+4. Validate each PR is accessible:
+
+```bash
+gh pr view "$PR_NUMBER" --repo "$REPO" --json number,title,url,state
+```
+
+If any PR cannot be resolved or accessed, fail immediately.
+
+#### Precheck 4 — Gather Operator Repo Context
+
+Gather context about the operator repository. Try the local clone first (faster, no API calls). If not available, fetch from the PR's GitHub repo via `gh api`. This is non-blocking -- if both fail, the skill falls back gracefully.
+
+```bash
+REPO_ROOT=$(git rev-parse --show-toplevel 2>/dev/null || echo "")
+OPERATOR_CONTEXT_SOURCE="none"
+
+# Strategy 1: Local clone (free, no API calls)
+if [ -n "$REPO_ROOT" ] && [ -f "$REPO_ROOT/go.mod" ]; then
+    GO_MODULE=$(head -1 "$REPO_ROOT/go.mod" | awk '{print $2}')
+
+    HAS_CR=false; HAS_LIBGO=false
+    grep -q "sigs.k8s.io/controller-runtime" "$REPO_ROOT/go.mod" && HAS_CR=true
+    grep -q "github.com/openshift/library-go" "$REPO_ROOT/go.mod" && HAS_LIBGO=true
+
+    if [ "$HAS_CR" = true ]; then FRAMEWORK="controller-runtime"
+    elif [ "$HAS_LIBGO" = true ]; then FRAMEWORK="library-go"
+    else FRAMEWORK="unknown"
+    fi
+
+    TEST_DIRS=$(find "$REPO_ROOT" -type d \( -name 'e2e' -o -name 'test' \) \
+        -not -path '*/vendor/*' 2>/dev/null | head -5)
+    HAS_MAKEFILE=$(test -f "$REPO_ROOT/Makefile" && echo "true" || echo "false")
+    OPERATOR_CONTEXT_SOURCE="local"
+
+# Strategy 2: Fetch from GitHub (costs 2-3 API calls)
+else
+    PR_HEAD_SHA=$(gh pr view "$PR_NUMBER" --repo "$REPO" --json headRefOid --jq '.headRefOid' 2>/dev/null)
+
+    GO_MOD_CONTENT=$(gh api "repos/$REPO/contents/go.mod?ref=$PR_HEAD_SHA" \
+        --jq '.content' 2>/dev/null | base64 -d 2>/dev/null || echo "")
+
+    if [ -n "$GO_MOD_CONTENT" ]; then
+        GO_MODULE=$(echo "$GO_MOD_CONTENT" | head -1 | awk '{print $2}')
+
+        HAS_CR=false; HAS_LIBGO=false
+        echo "$GO_MOD_CONTENT" | grep -q "sigs.k8s.io/controller-runtime" && HAS_CR=true
+        echo "$GO_MOD_CONTENT" | grep -q "github.com/openshift/library-go" && HAS_LIBGO=true
+
+        if [ "$HAS_CR" = true ]; then FRAMEWORK="controller-runtime"
+        elif [ "$HAS_LIBGO" = true ]; then FRAMEWORK="library-go"
+        else FRAMEWORK="unknown"
+        fi
+
+        HAS_MAKEFILE=$(gh api "repos/$REPO/contents/Makefile?ref=$PR_HEAD_SHA" \
+            --jq '.name' 2>/dev/null && echo "true" || echo "false")
+
+        TEST_DIRS=$(gh api "repos/$REPO/git/trees/$PR_HEAD_SHA?recursive=1" \
+            --jq '[.tree[] | select(.type=="tree") | select(.path | test("(^|/)e2e$|(^|/)test$"))] | .[0:5] | .[].path' \
+            2>/dev/null || echo "")
+
+        OPERATOR_CONTEXT_SOURCE="github"
+    else
+        GO_MODULE=""; FRAMEWORK="unknown"; TEST_DIRS=""; HAS_MAKEFILE="false"
+        echo "WARNING: Could not fetch operator repo context. Continuing without it."
+    fi
+fi
+
+echo "Operator context: source=$OPERATOR_CONTEXT_SOURCE module=$GO_MODULE framework=$FRAMEWORK makefile=$HAS_MAKEFILE"
+```
+
+#### Precheck 5 — Gather PR Change Context
+
+Fetch the list of changed files and a summary of the diff for each PR. This tells the skill what the PR actually changed, enabling error-to-file correlation and change-type classification.
+
+```bash
+for PR_NUMBER in "${PR_NUMBERS[@]}"; do
+    # Changed file paths (1 API call per PR)
+    PR_CHANGED_FILES=$(gh pr view "$PR_NUMBER" --repo "$REPO" \
+        --json files --jq '.files[].path')
+
+    # Classify change type from file paths
+    PR_HAS_API_CHANGES=$(echo "$PR_CHANGED_FILES" | grep -cE '(_types\.go|types_.*\.go)$' || true)
+    PR_HAS_CONTROLLER_CHANGES=$(echo "$PR_CHANGED_FILES" | grep -cE '(controller|reconcil).*\.go$' || true)
+    PR_HAS_TEST_CHANGES=$(echo "$PR_CHANGED_FILES" | grep -cE '_test\.go$' || true)
+    PR_HAS_CRD_CHANGES=$(echo "$PR_CHANGED_FILES" | grep -cE '(crd|crds)/.*\.yaml$' || true)
+    PR_HAS_RBAC_CHANGES=$(echo "$PR_CHANGED_FILES" | grep -cE 'rbac.*\.yaml$' || true)
+
+    echo "PR #$PR_NUMBER changes: files=$(echo "$PR_CHANGED_FILES" | wc -l)" \
+         "api=$PR_HAS_API_CHANGES controller=$PR_HAS_CONTROLLER_CHANGES" \
+         "test=$PR_HAS_TEST_CHANGES crd=$PR_HAS_CRD_CHANGES rbac=$PR_HAS_RBAC_CHANGES"
+done
+```
+
+**If ALL prechecks above passed, proceed to Phase 1.**
+**If ANY precheck FAILED (exit 1), STOP. Do NOT proceed further.**
+
+---
+
+### Phase 1-7: CI Monitoring, Analysis, and Fix Loop
+
+Load and execute the **ci-monitor skill** (`plugins/oape/skills/ci-monitor/SKILL.md`) for all subsequent phases. Pass the resolved context from Phase 0:
+
+- `REPO`, `PR_NUMBERS[]`, and all parsed flags
+- Operator repo context: `GO_MODULE`, `FRAMEWORK`, `TEST_DIRS`, `HAS_MAKEFILE`, `OPERATOR_CONTEXT_SOURCE`
+- PR change context: `PR_CHANGED_FILES`, change type counts (api, controller, test, crd, rbac)
+
+The skill handles:
+
+1. **Release Repo Discovery** -- fetches the ci-operator config from `openshift/release` for the target repo and branch. Parses job definitions (required/optional, fast/slow, cluster profile, release version) and resolves step registry references for failed jobs on demand.
+
+2. **SHA-Anchored Adaptive Polling** (Phase 1) -- records the HEAD SHA, polls GitHub checks and Prow commit statuses at adaptive intervals (60s/120s), detects SHA changes (clears stale results, waits settle period), and detects retests (timestamp comparison). Classifies jobs as fast/slow using ci-operator config when available, falling back to a name-based pattern table.
+
+3. **Failure Evidence Collection** (Phase 2) -- for each failed context, fetches GitHub Actions logs or Prow GCS artifacts (build-log, finished.json, JUnit XML, must-gather). When release context is available, maps failing steps to their step registry entries (container image, commands script).
+
+4. **Failure Classification** (Phase 3) -- classifies each failure into one of five modes: install (A), test (B), build (C), lint/boilerplate (D), or infra flake (E). Uses multi-signal detection (JUnit patterns, build-log regex, finished.json fields, ci-operator step metadata).
+
+5. **Deep Analysis and Flake Detection** (Phase 4) -- queries Sippy for historical pass rates using the OCP release version from the ci-operator config. Analyzes pass/fail sequences, error message consistency, and cluster health disruption patterns.
+
+6. **Stage-Aware Summary** (Phase 5) -- when exactly three PRs are provided, correlates failures across API / implementation / e2e stages and detects cross-stage dependencies.
+
+7. **Report Generation** (Phase 6) -- produces a structured markdown report including enriched job metadata (required/optional, cluster profile, step ref, commands) when release context is available.
+
+8. **Fix-Push-Rewatch Loop** (Phase 7) -- when `--max-fix-rounds > 0`, verifies the local branch matches the PR branch, applies fixes, runs local verification, pushes, and re-polls. Uses error signature hashing (normalized + SHA-256) to deterministically detect when a fix was ineffective (>= 75% hash match = same error = stop loop).
+
+---
+
+## Behavioral Rules
+
+1. **Collect everything first**: Never stop after the first failure. Gather evidence across all PRs and all failed jobs before producing the report.
+2. **No destructive operations**: Never propose force-push, branch deletion, or history rewriting.
+3. **Fix before retry**: Prefer deterministic fixes over blind retries. Only recommend `/retest` when evidence strongly suggests infra flake.
+4. **Explicit confidence**: Always state confidence level. If evidence is insufficient, say so and recommend deeper tools.
+5. **Stage-aware ordering**: When multiple PRs are involved, recommend fixing upstream PR failures first.
+6. **Budget-conscious**: Use adaptive polling to minimize API call consumption. Log the total calls made in the report footer.
+7. **Context-first**: Fetch release repo context before polling. Use ci-operator config for job classification when available. Fall back to name-based heuristics only when release context is unavailable.
+
+## Critical Failure Conditions
+
+Fail immediately if:
+1. No PR references are provided.
+2. `gh` or `curl` is missing or `gh` is unauthenticated.
+3. Repository cannot be resolved.
+4. A provided PR reference cannot be resolved to an accessible PR.
+
+## Exit Conditions
+
+- **Success**: All checks pass (possibly after fix rounds). Report produced.
+- **Partial Success**: Timeout reached or max-fix-rounds exhausted. Partial report produced with recommendations.
+- **Failure**: Precheck or resolution failure before monitoring begins.
diff --git a/plugins/oape/skills/ci-monitor/SKILL.md b/plugins/oape/skills/ci-monitor/SKILL.md
new file mode 100644
index 0000000..44752b3
--- /dev/null
+++ b/plugins/oape/skills/ci-monitor/SKILL.md
@@ -0,0 +1,994 @@
+---
+name: CI Monitor
+description: Monitor CI/Prow job status for OpenShift operator PRs with adaptive polling, context-aware failure analysis, and optional fix-push-rewatch loop
+---
+
+# CI Monitor Skill
+
+## Persona
+
+You are an **OpenShift CI/Prow monitoring specialist**. You monitor GitHub CI checks and Prow status contexts for pull requests, collect failure evidence, classify root causes, and optionally apply fixes. You think in terms of:
+
+- **Signal fidelity**: Distinguishing genuine failures from infra flakes, bot statuses, and transient errors
+- **Adaptive efficiency**: Minimizing GitHub API calls while never missing a state transition
+- **Prow internals**: ci-operator configs, step registry references, GCS artifact layouts, JUnit conventions
+- **Failure triage**: Classifying failures by mode (install, test, build, lint, infra) and mapping them to actionable fixes
+- **Flake awareness**: Cross-referencing test history via Sippy to distinguish regressions from known flakes
+- **Fix safety**: Only pushing fixes when confidence is high, the branch is correct, and the error is deterministically identified
+
+You are thorough (collect all failures before reporting), evidence-based (every classification cites log lines or JUnit entries), and budget-conscious (adaptive polling saves API calls).
+
+---
+
+## Release Repo Discovery
+
+Before polling begins, fetch the ci-operator configuration for the target repository from `openshift/release`. This provides authoritative job metadata that replaces name-based guessing.
+
+**Time budget**: This entire section (Steps 1-3) should complete in under **60 seconds**. If any step hangs or takes longer, skip it and fall back to name-based classification. Do NOT spend extended time parsing or re-fetching configs.
+
+### Step 1: Fetch ci-operator Config
+
+Fetch the config **exactly once** and save to a local temp file. Do NOT re-fetch from the API for subsequent parsing -- always read from the local file.
+
+```bash
+ORG=$(echo "$REPO" | cut -d'/' -f1)
+REPO_NAME=$(echo "$REPO" | cut -d'/' -f2)
+BRANCH=$(gh pr view "$PR_NUMBER" --repo "$REPO" --json baseRefName --jq '.baseRefName')
+
+CI_OP_CONFIG_PATH="ci-operator/config/$ORG/$REPO_NAME"
+CONFIG_FILE="$ORG-$REPO_NAME-$BRANCH.yaml"
+LOCAL_CONFIG="/tmp/ci-monitor-${ORG}-${REPO_NAME}-${BRANCH}-$(date +%s).yaml"
+
+# Fetch ONCE using raw.githubusercontent.com (faster, no base64 decoding needed)
+curl -sf "https://raw.githubusercontent.com/openshift/release/master/$CI_OP_CONFIG_PATH/$CONFIG_FILE" \
+    -o "$LOCAL_CONFIG" 2>/dev/null
+
+if [ ! -s "$LOCAL_CONFIG" ]; then
+    # Try master if branch-specific config not found
+    CONFIG_FILE="$ORG-$REPO_NAME-master.yaml"
+    curl -sf "https://raw.githubusercontent.com/openshift/release/master/$CI_OP_CONFIG_PATH/$CONFIG_FILE" \
+        -o "$LOCAL_CONFIG" 2>/dev/null
+fi
+
+if [ ! -s "$LOCAL_CONFIG" ]; then
+    echo "WARNING: No ci-operator config found for $REPO. Using name-based job classification."
+    USE_RELEASE_CONTEXT=false
+else
+    USE_RELEASE_CONTEXT=true
+    echo "Release config saved to $LOCAL_CONFIG"
+fi
+```
+
+**IMPORTANT**: All subsequent parsing in Steps 2 and 3 MUST read from `$LOCAL_CONFIG`, NOT from the GitHub API. Do not call `gh api` or `curl` for the same config file again.
+
+### Step 2: Parse Job Manifest
+
+Parse the locally saved config file (`$LOCAL_CONFIG`) to extract the job manifest. Do NOT fetch from GitHub again.
+
+Extract from the ci-operator config YAML a **job manifest** map with each test entry's properties:
+
+| Field | Source in Config | Use |
+|-------|-----------------|-----|
+| `job_name` | `tests[].as` | Match against Prow context names |
+| `job_type` | Derived: `cluster_profile` present = slow; `container` only = fast | Authoritative fast/slow classification |
+| `always_run` | `tests[].always_run` | Required vs conditional job |
+| `run_if_changed` | `tests[].run_if_changed` | Conditional trigger pattern |
+| `optional` | `tests[].optional` | Can fail without blocking merge |
+| `cluster_profile` | `tests[].steps.cluster_profile` | Cloud provider for infra-flake correlation |
+| `release_version` | `releases.latest.release.version` or `releases.latest.release.channel` | Sippy query parameter (resolves `` placeholder) |
+| `test_steps` | `tests[].steps.test[].ref` or `tests[].steps.test[].as` | Step registry references |
+| `commands` | `tests[].commands` | Direct command string (container tests) |
+| `credentials` | `tests[].steps.credentials[].name` and `.namespace` | Vault-injected secrets; trace auth failures here |
+
+### Step 3: Resolve Step Registry References
+
+For each `test_steps` reference, resolve to the actual commands using the step registry naming convention.
+
+The step registry maps ref names to directory paths by splitting on `-` at component boundaries. The directory path uses `/` separators, and files follow strict suffixes:
+
+| Component Type | Suffix | Example |
+|---------------|--------|---------|
+| Step ref | `-ref.yaml` | `openshift-e2e-test-ref.yaml` |
+| Commands script | `-commands.sh` | `openshift-e2e-test-commands.sh` |
+| Chain | `-chain.yaml` | `ipi-install-aws-chain.yaml` |
+| Workflow | `-workflow.yaml` | `ipi-aws-workflow.yaml` |
+
+Resolution procedure:
+
+```bash
+resolve_step_ref() {
+    local STEP_REF="$1"
+    # Convert ref name to directory path: openshift-e2e-test -> openshift/e2e/test
+    local STEP_DIR=$(echo "$STEP_REF" | sed 's|-|/|g')
+    local REG_BASE="ci-operator/step-registry"
+
+    # Fetch the ref YAML
+    local REF_YAML=$(gh api "repos/openshift/release/contents/$REG_BASE/$STEP_DIR/${STEP_REF}-ref.yaml" \
+        --jq '.content' 2>/dev/null | base64 -d 2>/dev/null || echo "")
+
+    if [ -z "$REF_YAML" ]; then
+        echo "STEP_REF_UNRESOLVED"
+        return
+    fi
+
+    # Extract image and commands file
+    local STEP_IMAGE=$(echo "$REF_YAML" | grep 'from:' | head -1 | awk '{print $2}')
+    local COMMANDS_FILE=$(echo "$REF_YAML" | grep 'commands:' | head -1 | awk '{print $2}')
+
+    # Fetch the actual commands script
+    local COMMANDS_SCRIPT=""
+    if [ -n "$COMMANDS_FILE" ]; then
+        COMMANDS_SCRIPT=$(gh api "repos/openshift/release/contents/$REG_BASE/$STEP_DIR/$COMMANDS_FILE" \
+            --jq '.content' 2>/dev/null | base64 -d 2>/dev/null || echo "")
+    fi
+
+    echo "IMAGE=$STEP_IMAGE"
+    echo "COMMANDS_FILE=$COMMANDS_FILE"
+    echo "SCRIPT_PREVIEW=$(echo "$COMMANDS_SCRIPT" | head -20)"
+}
+```
+
+Resolve step refs **on demand** after Phase 2 identifies failed jobs, not upfront for all jobs (saves API calls).
+
+### Graceful Fallback
+
+If `USE_RELEASE_CONTEXT=false`, all downstream phases fall back to name-based heuristics. The release context is an enrichment layer, not a hard dependency.
+
+---
+
+## Adaptive Polling Algorithm
+
+### Multi-PR Polling Strategy
+
+When monitoring multiple PRs, use a **single shared polling loop** that processes all PRs in each cycle. Do NOT poll PRs sequentially (that would triple cycle time for 3 PRs).
+
+Each poll cycle:
+
+```text
+1. For each active PR:
+   a. Check SHA (1 call)
+   b. Fetch statusCheckRollup (1 call)
+   c. Fetch commit statuses (1 call)
+   d. Update signals
+2. Determine interval from combined context state across all PRs
+3. Emit progress report
+4. Sleep interval
+= 3 calls per active PR per cycle
+```
+
+**Combined interval selection**: The polling interval is determined by the **slowest active PR**. If any active PR has slow contexts pending, use the slow interval. If all active PRs have only fast contexts pending, use the fast interval.
+
+**Early PR completion**: When all contexts for a PR become terminal, mark that PR as complete and skip its 3 API calls in subsequent cycles. This saves budget as PRs finish at different times.
+
+```text
+Example: monitoring PR #1, PR #2, PR #3
+  Cycle 1-5:   poll all 3 PRs (9 calls/cycle)
+  Cycle 6:     PR #1 complete, skip it (6 calls/cycle)
+  Cycle 7-12:  poll PR #2, PR #3 (6 calls/cycle)
+  Cycle 13:    PR #2 complete (3 calls/cycle)
+  Cycle 14-20: poll PR #3 only (3 calls/cycle)
+```
+
+**SHA changes are per-PR**: A SHA change on PR #1 clears signals and resets the timer for PR #1 only. PR #2 and PR #3 continue unaffected.
+
+### Record Initial State
+
+```bash
+for PR_NUMBER in "${PR_NUMBERS[@]}"; do
+    TRACKED_SHA[$PR_NUMBER]=$(gh pr view "$PR_NUMBER" --repo "$REPO" --json headRefOid --jq '.headRefOid')
+    SIGNALS[$PR_NUMBER]="{}"
+    PR_COMPLETE[$PR_NUMBER]=false
+done
+FIX_ATTEMPT=0
+ROUND_START=$(date +%s)
+```
+
+For each PR, maintain independently:
+- `tracked_sha`: the HEAD SHA being monitored
+- `signals`: map of context name to `{state, started_at, target_url, provider}`
+- `pr_complete`: whether all contexts are terminal (skip in future cycles)
+
+### Job Classification
+
+When `USE_RELEASE_CONTEXT=true`, classify jobs from the ci-operator config:
+
+| Config Property | Classification |
+|----------------|----------------|
+| `container:` present, no `cluster_profile` | **Fast** |
+| `cluster_profile:` present | **Slow** |
+| `steps:` with workflow referencing `ipi-*` or `upi-*` | **Slow** |
+
+When `USE_RELEASE_CONTEXT=false`, fall back to the name-based pattern table:
+
+| Tier | Patterns | Classification |
+|------|----------|----------------|
+| Slow (checked first) | `e2e`, `install`, `cluster`, `conformance`, `upgrade`, `aws`, `gcp`, `azure`, `metal`, `vsphere`, `libvirt`, `ovirt`, `openstack` | Slow |
+| Fast | `lint`, `vet`, `verify`, `verify-deps`, `unit`, `validate-boilerplate`, `coverage`, `go-build`, `images`, `bundle`, `shellcheck`, `gosec` | Fast |
+| Fallback | Any context not matching either tier | Fast (conservative: poll at 60s to avoid missing short jobs) |
+
+Slow patterns are checked first. If a context name matches both tiers (e.g., `test-e2e-aws`), the slow match wins.
+
+### Interval Selection
+
+After the first successful poll, select the polling interval:
+
+| Condition | Interval |
+|-----------|----------|
+| Any fast context still pending | **60s** (fast phase) |
+| Only slow contexts pending, running < 45 min | **120s** (slow phase) |
+| Only slow contexts pending, running >= 45 min | **60s** (finishing phase) |
+| All contexts terminal | Exit polling |
+
+### Auto-Adjusted Timeout
+
+After the first poll, auto-reduce timeout based on detected job types:
+
+| Detected Jobs | Timeout Cap |
+|---------------|-------------|
+| Only fast jobs (lint/verify/unit/build/images) | **30 min** |
+| e2e without cluster install | **60 min** |
+| e2e with cluster install | Full `--timeout-min` (default 120) |
+
+When `USE_RELEASE_CONTEXT=true`, this is authoritative (based on `cluster_profile` presence). When false, it is heuristic (based on name patterns).
+
+### SHA Change Detection
+
+On every poll iteration, for each active PR, before checking context states:
+
+```bash
+CURRENT_SHA=$(gh pr view "$PR_NUMBER" --repo "$REPO" --json headRefOid --jq '.headRefOid')
+
+if [ "$CURRENT_SHA" != "${TRACKED_SHA[$PR_NUMBER]}" ]; then
+    echo "[ci-monitor] SHA changed on PR #$PR_NUMBER: ${TRACKED_SHA[$PR_NUMBER]} -> $CURRENT_SHA"
+    TRACKED_SHA[$PR_NUMBER]="$CURRENT_SHA"
+    # Clear accumulated results for THIS PR only — they are stale
+    SIGNALS[$PR_NUMBER]="{}"
+    # Mark PR as active again (in case it was previously complete)
+    PR_COMPLETE[$PR_NUMBER]=false
+    # Wait for Prow to register new contexts
+    sleep "$SHA_SETTLE_SEC"
+    # Reset round timer for this PR
+    ROUND_START=$(date +%s)
+    continue
+fi
+```
+
+**Why 90s settle**: After a push, Prow needs 30-60s to register new status contexts. Polling immediately would see "0 pending, 0 failed" and incorrectly conclude everything passed.
+
+### Retest Detection (no SHA change)
+
+When `/retest` or `/test ` is commented, the SHA stays the same but Prow creates new job runs. Detect by comparing `started_at` timestamps:
+
+```bash
+for CONTEXT_NAME in $(echo "$CURRENT_CONTEXTS" | jq -r '.[].name'); do
+    PREV_STATE=$(echo "$SIGNALS" | jq -r --arg n "$CONTEXT_NAME" '.[$n].state // empty')
+    PREV_STARTED=$(echo "$SIGNALS" | jq -r --arg n "$CONTEXT_NAME" '.[$n].started_at // empty')
+    CURR_STATE=$(echo "$CURRENT_CONTEXTS" | jq -r --arg n "$CONTEXT_NAME" \
+        '.[] | select(.name==$n) | .state')
+    CURR_STARTED=$(echo "$CURRENT_CONTEXTS" | jq -r --arg n "$CONTEXT_NAME" \
+        '.[] | select(.name==$n) | .started_at')
+
+    if echo "failure success cancelled timed_out" | grep -qw "$PREV_STATE"; then
+        if echo "pending queued" | grep -qw "$CURR_STATE"; then
+            echo "Retest detected: $CONTEXT_NAME restarted (terminal -> pending)"
+            # Reset signal to pending
+        elif [ -n "$CURR_STARTED" ] && [ -n "$PREV_STARTED" ] && \
+             [ "$CURR_STARTED" \> "$PREV_STARTED" ]; then
+            echo "Rerun detected: $CONTEXT_NAME completed a new run"
+            # Update signal with new result
+        fi
+    fi
+done
+```
+
+### Unified Signal Tracking
+
+Merge both Checks API and commit status contexts into a single list.
+
+**How to fetch signals** (use these specific commands to avoid `gh` CLI field errors):
+
+```bash
+# GitHub Checks (via GraphQL statusCheckRollup) — works reliably
+gh pr view "$PR_NUMBER" --repo "$REPO" --json statusCheckRollup \
+    --jq '.statusCheckRollup[] | "\(.name)\t\(.state)\t\(.detailsUrl)"'
+
+# Prow commit statuses (via REST API) — more reliable than gh pr checks
+gh api "repos/$REPO/commits/$TRACKED_SHA/status" \
+    --jq '.statuses[] | "\(.context)\t\(.state)\t\(.target_url)"'
+```
+
+Do NOT use `gh pr checks --json status,conclusion` -- the `status` field is not available in all `gh` versions and will error with "Unknown JSON field."
+
+For each entry track:
+- `name` / `context`
+- `state` (`queued`, `in_progress`, `pending`, `success`, `failure`, `cancelled`, `timed_out`, `action_required`, `skipped`)
+- `provider` (`github-actions`, `github-check`, `prow-status-context`)
+- `target_url` / `details_url`
+- `started_at`
+
+### Non-Test Context Exclusions
+
+Exclude or handle non-CI contexts separately. Do not count these as failures:
+
+| Context Pattern | Classification | Action |
+|-----------------|---------------|--------|
+| `tide` | Merge gate | Report description (e.g., "needs: lgtm, approved"), do not treat as failure |
+| `CodeRabbit` | Review bot | Ignore entirely |
+| `Mergeable` | GitHub merge check | Report status, do not count as CI failure |
+| `DCO` | Commit signing | Report status, do not count as CI failure |
+| `stale` | Staleness bot | Ignore entirely |
+
+### User-Defined Context Exclusions (`--ignore-context`)
+
+When `--ignore-context ` is provided (one or more times), any CI context whose name contains the pattern (case-insensitive substring match) is treated the same as `CodeRabbit` or `stale` — **ignored entirely**. The context is:
+- Excluded from the pending/running count (does not delay termination)
+- Excluded from the failure count (does not affect the verdict)
+- Not included in the Prow Job Breakdown table
+- Not fetched for evidence collection
+
+This is primarily used when the ci-monitor runs as a CI job itself, to prevent it from watching its own check (e.g., `--ignore-context oape-ci-monitor`).
+
+When `USE_RELEASE_CONTEXT=true`, also check `tests[].optional: true` from the job manifest. Optional jobs that fail are reported but do not block the overall verdict.
+
+**Severity labeling for optional jobs**: If a job is marked `optional: true`, NEVER label its failure as "Blocker" or "Critical Severity." Label it as "Non-blocking (optional)" regardless of the failure mode. Optional job failures should not change the PR's overall verdict from PASS to FAIL.
+
+### Progress Reporting
+
+Emit status updates during polling so the user has visibility into long monitoring sessions.
+
+**Per-poll one-liner** (after every poll iteration):
+
+```text
+[ci-monitor] 12:34 | PR #342 | 8/12 complete | 3 pending (e2e-aws, e2e-gcp, upgrade) | 1 failed (lint) | elapsed: 23m | next poll: 120s
+```
+
+**Milestone summary** (every 10 minutes):
+
+```text
+[ci-monitor] === 30m checkpoint ===
+  PR #342: 10/12 complete | 2 pending: e2e-aws (slow, ~15m est.), upgrade (slow, ~20m est.)
+  PR #343: 6/6 complete | ALL PASSED
+  PR #344: 4/8 complete | 1 failed: unit | 3 pending
+  API calls so far: 180 | Rate limit remaining: 4,820
+  Auto-retests posted: 0/2
+```
+
+**Event-driven reports** (immediately when detected):
+
+```text
+[ci-monitor] SHA changed on PR #342 (abc1234 -> def5678). Clearing stale results, waiting 90s...
+[ci-monitor] Retest detected: e2e-aws restarted on PR #342 (terminal -> pending)
+[ci-monitor] Auto-retest posted on PR #344: all 2 failures are infra flakes. Retest 1/2.
+[ci-monitor] PR #343 complete: ALL PASSED (6/6)
+```
+
+Track for milestone summaries:
+- `POLL_COUNT`: total poll iterations
+- `API_CALL_COUNT`: running total of GitHub API calls
+- `LAST_MILESTONE`: timestamp of last milestone report (emit every 10 min)
+
+### Termination Conditions
+
+Stop polling a PR when:
+1. No checks AND no status contexts are `pending`, `in_progress`, or `queued`.
+2. Timeout reached -- mark unresolved signals as `timed_out`.
+3. PR state changed to `CLOSED` or `MERGED`.
+
+---
+
+## Evidence Collection Procedures
+
+For every signal in a terminal failure state (`failure`, `cancelled`, `timed_out`), gather evidence using the strategy appropriate to its provider.
+
+**CRITICAL RULES to prevent loops**:
+- Fetch each artifact **exactly once**. Do NOT retry failed downloads. If a `curl` fails, mark as `partial` and move on.
+- Do NOT re-fetch an artifact you already have. Before fetching, check if the local file already exists.
+- **Time budget**: Evidence collection for ALL failed jobs combined should complete in under **3 minutes**. If it takes longer, stop collecting and proceed to classification with whatever evidence you have.
+- Save all artifacts to a unique session directory: `/tmp/ci-monitor-$PR_NUMBER-$(date +%s)/`
+- Process each failed context ONCE in a single pass -- do NOT loop back to re-process contexts.
+
+### GitHub Actions Failures
+
+Extract the run ID from the details URL and fetch failed logs:
+
+```bash
+gh run view "$RUN_ID" --repo "$REPO" --log-failed
+```
+
+### Prow Status Context Failures
+
+For each failed `ci/prow/*` context:
+
+1. **Parse the Prow job URL** from `target_url`. Extract:
+   - GCS bucket path (e.g., `gs/test-platform-results/pr-logs/pull/.../`)
+   - Job name (e.g., `pull-ci-openshift-must-gather-operator-master-validate-boilerplate`)
+   - Build ID
+
+2. **Derive artifact base URL** from the `target_url`:
+
+```bash
+ARTIFACT_BASE=$(echo "$TARGET_URL" | sed 's|https://prow.ci.openshift.org/view/gs/|https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/|')
+```
+
+3. **Download key artifacts in a single pass** (skip in `--fast` mode except `build-log.txt`):
+
+For each failed Prow context, run ONE batch of downloads. Do NOT loop or retry:
+
+```bash
+ARTIFACT_DIR="/tmp/ci-monitor-$PR_NUMBER-$(date +%s)/$JOB_NAME"
+mkdir -p "$ARTIFACT_DIR"
+
+# Download all artifacts in one pass — no retries
+curl -sf "$ARTIFACT_BASE/build-log.txt" -o "$ARTIFACT_DIR/build-log.txt" 2>/dev/null || true
+curl -sf "$ARTIFACT_BASE/finished.json" -o "$ARTIFACT_DIR/finished.json" 2>/dev/null || true
+
+# Skip these in --fast mode
+if [ "$FAST_MODE" != true ]; then
+    curl -sf "$ARTIFACT_BASE/prowjob.json" -o "$ARTIFACT_DIR/prowjob.json" 2>/dev/null || true
+fi
+
+# Mark which artifacts were obtained
+echo "Artifacts: $(ls "$ARTIFACT_DIR" 2>/dev/null | tr '\n' ', ')"
+```
+
+| Artifact | Path | Purpose |
+|----------|------|---------|
+| `build-log.txt` | `$ARTIFACT_BASE/build-log.txt` | Primary CI log -- always fetch |
+| `finished.json` | `$ARTIFACT_BASE/finished.json` | Exit code, timestamps, result |
+| `prowjob.json` | `$ARTIFACT_BASE/prowjob.json` | Full Prow job spec (skip in fast mode) |
+| `junit*.xml` | `$ARTIFACT_BASE/artifacts//junit*.xml` | Per-test pass/fail with error messages (skip in fast mode) |
+4. **Parse JUnit XML** (unless `--fast`):
+   - Enumerate every `` with a `` or `` child.
+   - Extract: test name, class name, failure message, stack trace snippet (first 40 lines).
+   - Count total tests, passed, failed, skipped, errored.
+
+5. **Detect must-gather availability** (unless `--fast`):
+   - MUST check for jobs that provision clusters (Mode A install failures or Mode B e2e test failures with `cluster_profile` present in the job manifest).
+   - Skip for lint, unit, build, and container-only jobs -- they never produce must-gather artifacts.
+   - Path: `$ARTIFACT_BASE/artifacts/*/must-gather.tar`
+   - Check availability:
+     ```bash
+     MUST_GATHER_URL="$ARTIFACT_BASE/artifacts/must-gather/must-gather.tar"
+     if curl -sf --head "$MUST_GATHER_URL" >/dev/null 2>&1; then
+         echo "must-gather available: $MUST_GATHER_URL"
+     fi
+     ```
+   - If present, include the download URL in the report's Evidence section.
+   - If absent, note "must-gather: not available" in the report.
+
+6. If any artifact fetch fails, mark evidence as `partial` and **move on immediately**. Do NOT retry. Do NOT re-fetch. The `|| true` in the download commands ensures failures are silent and non-blocking.
+
+### On-Demand PR Diff Fetch
+
+When a failure is identified and the failing file is in `PR_CHANGED_FILES`, fetch the diff for that specific file to correlate the error with the actual code change:
+
+```bash
+FAILING_FILE="pkg/controller/foo.go"
+if echo "$PR_CHANGED_FILES" | grep -qx "$FAILING_FILE"; then
+    FILE_DIFF=$(gh pr diff "$PR_NUMBER" --repo "$REPO" -- "$FAILING_FILE" 2>/dev/null || echo "")
+fi
+```
+
+This is fetched **per failing file, on demand** -- not upfront for the entire PR. Include the relevant diff excerpt (max 30 lines around the error) in the report's Evidence section.
+
+### Step-Level Failure Mapping (when release context available)
+
+When `USE_RELEASE_CONTEXT=true` and the failed job uses multi-stage steps:
+
+1. Identify the failing step name from the build log (look for `step "" failed`).
+2. Resolve the step ref using the step registry (see Release Repo Discovery, Step 3).
+3. Record the step's image, commands file, and script preview in the evidence.
+
+This maps a generic "e2e-aws failed" to "step `openshift-e2e-test` failed, which runs `openshift-tests run openshift/conformance/parallel` in the `tests` image."
+
+---
+
+## Failure Classification Rules
+
+For each failed job, classify into one of the following failure modes. Apply rules in order -- first match wins.
+
+### Mode A: Install Failure
+
+**Detection** (match ANY):
+- JUnit contains a test matching `install should succeed:*`
+- JUnit test names containing `cluster-install`, `bootstrap`, `infrastructure-setup`, or `infra-setup`
+- Build log matches `level=fatal.*installer`, `cluster creation failed`, `bootstrap.*timed out`, or `waiting for bootstrapComplete`
+- `finished.json` has `result: "ABORTED"` with install-stage timestamps
+- When release context available: the failing step ref is part of a `pre:` chain in the workflow (install phase)
+
+**Analysis focus**: Identify install stage, extract installer errors.
+
+### Mode B: Test Failure (e2e, unit, integration)
+
+**Detection**: JUnit contains failed `` entries that are NOT install tests (Mode A).
+
+**Analysis focus**: List failed tests, classify isolation/co-failure/mass-failure patterns. When release context available, include the test step ref and its commands.
+
+### Mode C: Build / Compile Failure
+
+**Detection** (match ANY):
+- Build log contains Go compile errors (`cannot find package`, `undefined:`, `syntax error`, `imported and not used`)
+- Build log contains `make: *** [...] Error` with compile-related targets
+- No JUnit output present (build failed before tests ran)
+
+**Analysis focus**: Extract compiler errors. When operator repo context available, map errors to local Go packages via `GO_MODULE`. When PR change context available, check if the failing file is in `PR_CHANGED_FILES` -- if yes, fetch the diff for that file on demand and correlate the error line with the actual code change:
+
+```bash
+# Fetch diff only for the specific failing file
+gh pr diff "$PR_NUMBER" --repo "$REPO" -- "$FAILING_FILE"
+```
+
+If the error is in a file NOT in `PR_CHANGED_FILES`, it may be a dependency or generated-code issue (e.g., `zz_generated.deepcopy.go` not regenerated after types changed).
+
+### Mode D: Lint / Static Analysis / Boilerplate
+
+**Detection** (match ANY):
+- Job name contains `lint`, `vet`, `verify`, `validate-boilerplate`, `verify-deps`
+- When release context available: job's `commands` field contains `make lint`, `make verify`, `make vet`
+
+**Analysis focus**: Extract lint/validation errors, distinguish CI image issues from actual drift.
+
+### Mode E: CI Infrastructure / Transient
+
+**Detection** (match ANY of these patterns in build log, with no test-level errors):
+- `registry.ci.openshift.org.*timeout` or `registry.ci.openshift.org.*error`
+- `ImagePullBackOff`, `ErrImagePull`, `ImagePullErr`
+- `i/o timeout`, `connection refused`, `dial tcp.*timeout`
+- `etcdserver: request timed out`, `lease lost`
+- `error creating.*instance`, `quota exceeded`, `InsufficientInstanceCapacity`
+- `unable to get lease`, `failed to acquire lease`
+- `context deadline exceeded` with no test failures
+- Error in Prow infrastructure (pod scheduling, volume mount) rather than in test code
+
+**Analysis focus**: Mark as `probable-infra-flake`. If auto-retest is enabled, post `/retest` automatically (see Auto-Retest Protocol below). Otherwise, recommend `/retest` in the report.
+
+---
+
+## Auto-Retest Protocol
+
+When `--no-auto-retest` is NOT set and Mode E (infra flake) is detected, automatically post a `/retest` comment to re-trigger failed jobs.
+
+### Eligibility
+
+All conditions must be met:
+- ALL failures on the PR are Mode E (infra flake). If any failure is Mode A/B/C/D, do not auto-retest -- fix the real failure first.
+- The failing context is not optional (`tests[].optional != true` when release context available).
+- `RETEST_COUNT < 2` for this monitoring session (hard cap to prevent infinite retest loops).
+
+### Procedure
+
+```bash
+RETEST_COUNT=0
+MAX_AUTO_RETESTS=2
+
+# After Phase 3 classifies all failures for a PR:
+ALL_MODE_E=true
+for FAILURE in $(pr_failures "$PR_NUMBER"); do
+    if [ "$(get_failure_mode "$FAILURE")" != "E" ]; then
+        ALL_MODE_E=false
+        break
+    fi
+done
+
+if [ "$ALL_MODE_E" = true ] && [ "$RETEST_COUNT" -lt "$MAX_AUTO_RETESTS" ]; then
+    echo "[ci-monitor] All failures on PR #$PR_NUMBER are infra flakes. Posting /retest..."
+    gh pr comment "$PR_NUMBER" --repo "$REPO" --body "/retest"
+    RETEST_COUNT=$((RETEST_COUNT + 1))
+    echo "[ci-monitor] Auto-retest $RETEST_COUNT/$MAX_AUTO_RETESTS posted. Resuming polling..."
+    # Polling loop naturally handles re-poll: contexts go back to pending
+fi
+```
+
+### Guardrails
+
+- **Max 2 auto-retests** per monitoring session. After 2 retests, if infra flakes persist, report them and stop.
+- **Only when ALL failures are Mode E**. A single real failure (Mode A/B/C/D) disables auto-retest for that PR.
+- **Logged in final report**: each auto-retest is recorded with timestamp, affected contexts, and outcome.
+- **Disabled with `--no-auto-retest`**: users can opt out entirely.
+
+### Interaction with Fix Loop
+
+Auto-retest and the fix loop serve different purposes and apply to different failure modes. Their ordering:
+
+1. Polling completes (all contexts terminal).
+2. Evidence collection (Phase 2) runs for all failed contexts.
+3. Failure classification (Phase 3) assigns a mode to each failure.
+4. **Auto-retest evaluated first**: if ALL failures on a PR are Mode E, post `/retest` and resume polling. The fix loop is NOT entered.
+5. **Fix loop evaluated second**: if any failures are Mode B/C/D (and auto-retest did not trigger), enter the fix loop.
+
+If auto-retest triggers and the retry produces a new real failure (Mode B/C/D), the next classification round will skip auto-retest (not all failures are Mode E anymore) and enter the fix loop instead.
+
+---
+
+## Deep Analysis Protocol
+
+### Sippy Historical Pass Rate Lookup (MANDATORY for test failures)
+
+When any test failure (Mode B) is detected, you MUST query Sippy for historical pass rates. Do NOT skip this step. Do NOT report "Sippy queries: 0 (skipped)" when a release version is available.
+
+```bash
+RELEASE_VERSION="$RESOLVED_RELEASE"
+curl -sf "https://sippy.dptools.openshift.org/api/tests?release=$RELEASE_VERSION&filter.test_name=$TEST_NAME"
+```
+
+**How to resolve `RELEASE_VERSION`** (try in order):
+1. From ci-operator config: `releases.latest.release.version` or `releases.latest.integration.name` (e.g., `"5.0"`)
+2. From Prow job name: extract version pattern (e.g., `4.15` from `pull-ci-...-4.15-e2e-aws`)
+3. If both fail: log a warning but still attempt the query with a reasonable default (latest GA release)
+
+When `USE_RELEASE_CONTEXT=true`, the release version is already extracted in the Release Repo Discovery step. Use it directly.
+
+Classification:
+- Pass rate >= 95% and now failing -> **likely genuine regression**
+- Pass rate < 95% with `open_bugs > 0` -> **known flaky test**
+- Pass rate < 95% with `open_bugs == 0` -> **unstable test**
+
+### Prow Job History / Pass Sequence Analysis
+
+Classify pass sequence pattern (left = newest):
+- `FFFFFFFFFF` -> Permafail (High priority)
+- `FFFFSSSSSS` -> Recent regression (High)
+- `SSSSSFFFFF` -> Resolved (Low)
+- `SFSFSFSFSF` -> Flaky (Medium)
+
+### Failure Output Consistency
+
+Compare error messages across multiple failures:
+- **Highly consistent** (>90%): single cascading root cause
+- **Moderately consistent** (50-90%): primary issue with secondaries
+- **Inconsistent** (<50%): multiple issues or environmental instability
+
+### Disruption / Cluster Health Correlation (e2e jobs only, unless `--fast`)
+
+Check for cluster-level disruption: `ci-cluster-network-liveness` failures, operator degradation, etcd issues.
+
+---
+
+## Root Cause Tracing Protocol
+
+For each failure, trace the root cause to its origin by reasoning through the available context layers. Do not use a fixed lookup table. Instead, follow this diagnostic decision tree and **cite the evidence at each step** so the reasoning is verifiable.
+
+### Step 1: Does the error reference a specific file?
+
+If the error message or stack trace points to a source file:
+
+- Is that file in `PR_CHANGED_FILES`?
+  - **Yes** → The PR likely introduced this issue. Fetch the on-demand diff for this file and correlate the error line with the change.
+  - **No** → Is the file in the operator repo (from operator context)?
+    - **Yes** → The issue is in pre-existing code, not caused by this PR.
+    - **No** → Is the file from a CI step container or build image? Check the step registry ref if available.
+
+### Step 2: Is the error about a missing tool, command, or image?
+
+If the error contains "command not found", "executable not found", "image not found", or similar:
+
+- Check the ci-operator config: what `container.from` or step `from:` image is used for this job?
+- Is the missing tool expected to be in that image?
+- Trace to: the ci-operator config (which image is specified) and optionally the Dockerfile that builds it.
+
+### Step 3: Is the error about authentication, credentials, or secrets?
+
+If the error contains "authentication failed", "unauthorized", "password is incorrect", "token expired", or similar:
+
+- Check the ci-operator config: does this job declare `credentials` entries?
+- If yes, identify the credential name and namespace. The credential itself is stored externally (typically Vault) and injected by Prow.
+- The fix is NOT in the code -- it's in the secret store or the ci-operator credential declaration.
+
+### Step 4: Is the error transient or environmental?
+
+If the error matches infrastructure patterns (network timeout, quota exceeded, lease exhaustion, registry errors):
+
+- No code or config fix is needed. The issue is in the CI environment or cloud provider.
+- Recommend `/retest`.
+
+### Step 5: Is the error about missing generated code?
+
+If the build error references `zz_generated.deepcopy.go`, `zz_generated.defaults.go`, or CRD YAML files:
+
+- Check `PR_CHANGED_FILES`: were `_types.go` files changed?
+- Were the generated files also updated?
+- If types were changed but generated files were not → incomplete code generation. PR author needs to run `make generate && make manifests`.
+
+### Step 6: None of the above matched
+
+If the failure doesn't fit any of the patterns above, report it with:
+- All available evidence (log excerpt, JUnit entry, step info)
+- The context layers that were checked and what they showed
+- Confidence: low
+- A recommendation for manual investigation
+
+### Output Format (MANDATORY)
+
+Every failure in the report MUST include a **Root Cause Trace**. Do NOT skip this section. Do NOT replace it with a free-form "Root Cause" paragraph. Use this exact structure:
+
+```text
+Root Cause Trace:
+  1. 
+  2. 
+  3. 
+  ...
+  Fix location: 
+  Fix owner: ) | credential owner | transient — /retest>
+  Confidence: 
+```
+
+Rules:
+- Every step in the trace MUST reference a concrete artifact (log line number, file path, config entry, PR change list result)
+- Do NOT assert a location without citing what led to that conclusion
+- If a failure involves credentials from the ci-operator config, explicitly state the credential name, namespace, and that it is declared in `openshift/release` (not in the PR's code)
+- If a job is marked `optional: true` in the ci-operator config, do NOT label it as a "Blocker" -- label it as "Non-blocking (optional)"
+
+---
+
+## Stage-Aware Summary Logic
+
+If exactly three PRs were provided, summarize by stage:
+- **PR #1 (API)**: schema/codegen/validation risks, boilerplate/generation drift
+- **PR #2 (Implementation)**: controller logic, build, unit test, RBAC consistency
+- **PR #3 (E2E)**: scenario coverage, e2e environment, cluster install stability
+
+When PR change context is available, validate staging:
+- PR #1 should primarily contain API type changes (`_types.go`, CRD YAML). Flag if it contains controller changes.
+- PR #2 should primarily contain controller/reconciler changes. Flag if it modifies API types (should be in PR #1).
+- PR #3 should primarily contain test files (`_test.go`, `e2e/`). Flag if it contains production code.
+
+Cross-stage dependency detection:
+- PR #2 compile failure referencing PR #1 types -> fix PR #1 first
+- PR #3 "CRD not found" -> PRs #1/#2 not merged yet
+- PR #2 build error in a file from `PR_CHANGED_FILES` -> directly caused by this PR
+- PR #2 build error in a file NOT in `PR_CHANGED_FILES` -> likely dependency on PR #1 or generated code drift
+
+---
+
+## Report Template
+
+Return a structured markdown report. When release context is available, include enriched job metadata.
+
+```text
+=== CI Monitor Report ===
+
+Repository: 
+Go Module:  | Framework:        (when operator context available)
+PR Head SHA: 
+Monitoring: adaptive polling (60s/120s/60s) | Timeout: m
+Fix Round:  of  | SHA Changes Detected: 
+Signals Observed:  checks,  status contexts
+Release Context: 
+OCP Release:                                  (when release context available)
+Mode: 
+
+────────────────────────────────────────
+PR Results
+────────────────────────────────────────
+PR # "" — PASS | FAIL | TIMED_OUT
+  Checks:  <pass>/<total> passed
+  Prow:    <pass>/<total> passed
+  Failed:  <list of failed context names>
+
+────────────────────────────────────────
+Failure Analysis
+────────────────────────────────────────
+1) [PR #<n>] <check-name>
+   Provider: <github-actions | prow-status-context>
+   Failure Mode: <install | test | build | lint/boilerplate | infra-flake>
+   Required: <yes | no (optional)>                     (when release context available)
+   Cluster: <aws | gcp | azure | none>                 (when release context available)
+   Step: <step-ref-name>                                (when release context available)
+   Commands: <actual command or script>                 (when release context available)
+   Job URL: <prow view url or actions url>
+   Artifacts: <gcsweb artifact browser url>
+
+   Evidence:
+     <key log excerpt — max 20 lines>
+
+   JUnit Summary (if available):
+     Total: <N> | Passed: <N> | Failed: <N> | Skipped: <N>
+
+   Sippy Flake Check (if available):
+     - <test name>: pass_rate=<N>% trend=<dir> open_bugs=<N>
+
+   Root Cause Trace:
+     1. <evidence point — what the error says>
+     2. <evidence point — where it originates (PR code? repo? CI config? Vault?)>
+     3. <evidence point — what context layers confirm>
+     Fix location: <specific file, config, or system>
+     Fix owner: <who can make the change>
+
+   Root Cause Hypothesis: <text>
+   Confidence: <high | medium | low>
+   Fixable by agent: <yes | no — reason>
+   Error Signature: <hash>                              (for fix loop tracking)
+
+   Suggested Fixes:
+     1. <most targeted fix>
+     2. <alternative>
+
+   Validation:
+     - <command to verify fix locally>
+     - <command to rerun CI>
+
+────────────────────────────────────────
+Prow Job Breakdown
+────────────────────────────────────────
+| Context | State | Mode | Required | Flake? | Action |
+|---|---|---|---|---|---|
+| ci/prow/<name> | failure | test | yes | no (98%) | fix required |
+| ci/prow/<name> | failure | infra | yes | yes (72%) | /retest |
+| ci/prow/<name> | success | — | no (optional) | — | — |
+| tide | pending | gate | — | — | needs: lgtm, approved |
+
+────────────────────────────────────────
+Recommended Next Actions
+────────────────────────────────────────
+1. <highest priority fix>
+2. <second action>
+3. <rerun plan>
+
+────────────────────────────────────────
+Auto-Retest Log                                        (when auto-retest triggered)
+────────────────────────────────────────
+| # | PR | Timestamp | Contexts Retested | Outcome |
+|---|---|---|---|---|
+| 1 | #342 | 12:45 | e2e-aws, e2e-gcp | passed on retry |
+| 2 | #342 | 13:10 | upgrade | still failing (infra) |
+
+────────────────────────────────────────
+API Budget
+────────────────────────────────────────
+Total GitHub API calls: <N>
+Release context calls: <N>
+PR change context calls: <N>
+Auto-retest comment calls: <N>
+Polling rounds: <N>
+```
+
+### Post Report as PR Comment (`--post-comment`)
+
+When `--post-comment` is set, after generating the report above, post it as a GitHub PR comment for each monitored PR:
+
+```bash
+# Only post if gh is authenticated and we have a PR number
+if [ "$POST_COMMENT" = true ]; then
+    for PR_NUMBER in "${PR_NUMBERS[@]}"; do
+        gh pr comment "$PR_NUMBER" --repo "$REPO" --body "$REPORT"
+        echo "[ci-monitor] Posted report as comment on PR #$PR_NUMBER"
+    done
+fi
+```
+
+The full report (from `=== CI Monitor Report ===` through `API Budget`) is posted as the comment body. GitHub renders the markdown natively.
+
+If the comment exceeds GitHub's 65536 character limit, truncate the Evidence sections (keeping the Root Cause Trace and Suggested Fix for each job) and append a note: `_Report truncated. See full output in CI job logs._`
+
+---
+
+## Fix-Push-Rewatch Protocol
+
+When `--max-fix-rounds > 0` and the analysis identifies fixable failures.
+
+### Eligibility
+
+Only attempt auto-fix when ALL conditions are met:
+- Failure Mode is B (test), C (build), or D (lint/boilerplate)
+- Root cause hypothesis has `high` or `medium` confidence
+- The fix can be applied to files in the current working directory
+- When operator repo context available: the fix targets files within the detected framework pattern
+
+### Branch Verification
+
+Before applying any fix, verify the local checkout matches the PR branch:
+
+```bash
+PR_BRANCH=$(gh pr view "$PR_NUMBER" --repo "$REPO" --json headRefName --jq '.headRefName')
+CURRENT_BRANCH=$(git branch --show-current)
+
+if [ "$CURRENT_BRANCH" != "$PR_BRANCH" ]; then
+    echo "Switching to PR branch: $PR_BRANCH"
+    git fetch origin "$PR_BRANCH"
+    git checkout "$PR_BRANCH"
+    git pull origin "$PR_BRANCH"
+fi
+```
+
+If the branch cannot be checked out (not a local clone, permission issues), skip the fix loop and produce a report-only output.
+
+### Apply Fix
+
+1. Make the code change based on the suggested fix.
+2. Run local verification. When operator repo context available, use discovered capabilities:
+   - If `HAS_MAKEFILE=true` and Makefile has `verify` target: `make verify`
+   - Always: `go build ./...`, `go vet ./...`
+   - For Mode D (lint): `make lint` or `make verify` depending on what the job runs (from release context `commands` field)
+3. If local verification fails, revert and report without pushing.
+
+### Commit and Push
+
+```bash
+git add -A
+git commit -m "fix: <description of CI fix>"
+git push
+```
+
+### Error Signature Tracking
+
+To determine if a fix attempt resolved the issue, track error signatures across rounds:
+
+1. **Extract error signature** from each failed context: take the first meaningful error line from the build log or JUnit failure message.
+2. **Normalize**: strip timestamps, line numbers, hex addresses (`0x[a-f0-9]+`), UUIDs (`[a-f0-9-]{36}`), temp file paths (`/tmp/[^ ]+`).
+3. **Hash**: `echo "$NORMALIZED_ERROR" | sha256sum | cut -c1-16`
+4. **Store**: maintain a map of `context_name -> error_hash` per fix round.
+
+### Same-Error Detection
+
+After each fix round completes and new failures are collected:
+
+```bash
+SAME_COUNT=0
+TOTAL_FAILED=0
+for CONTEXT in $(failed_contexts); do
+    TOTAL_FAILED=$((TOTAL_FAILED + 1))
+    PREV_HASH=$(get_prev_round_hash "$CONTEXT")
+    CURR_HASH=$(get_curr_round_hash "$CONTEXT")
+    if [ "$PREV_HASH" = "$CURR_HASH" ] && [ -n "$PREV_HASH" ]; then
+        SAME_COUNT=$((SAME_COUNT + 1))
+    fi
+done
+
+SAME_RATIO=$((SAME_COUNT * 100 / TOTAL_FAILED))
+if [ "$SAME_RATIO" -ge 75 ]; then
+    echo "Fix ineffective: $SAME_RATIO% of failures have identical error signatures."
+    echo "Stopping fix loop."
+    # Produce final report
+fi
+```
+
+If >= 75% of failed contexts share the same error hash as the previous round, classify as "same error" and stop the fix loop.
+
+### Termination
+
+- `FIX_ATTEMPT >= max-fix-rounds`: stop, produce final report with all rounds summarized
+- New round passes: report success
+- New round fails with SAME error (>= 75% hash match): stop, fix didn't work
+- New round fails with a DIFFERENT error: attempt another fix if rounds remain
+
+### Never Auto-Fix
+
+- Install failures (Mode A) -- require cluster-level investigation
+- Infra flakes (Mode E) -- recommend `/retest` only
+- Repo-wide failures (same error across all open PRs) -- not caused by this PR
+- Low-confidence hypotheses -- report only, let user decide
+
+---
+
+## Integration Notes
+
+This skill is invoked by the `/oape:ci-monitor` command and follows this flow:
+
+1. Command validates inputs and resolves PRs (Phase 0)
+2. Command gathers operator repo context (Phase 0, Precheck 4)
+3. Command gathers PR change context (Phase 0, Precheck 5)
+4. Skill fetches release repo context (ci-operator config + step registry)
+5. Skill runs adaptive polling with SHA/retest tracking
+6. Skill collects failure evidence (GCS artifacts, JUnit, logs, on-demand diffs)
+7. Skill classifies failure modes using config-aware rules
+8. Skill performs deep analysis (Sippy, history, consistency)
+9. Skill traces root cause for each failure through context layers (PR changes, repo, release config, infrastructure)
+10. Skill produces stage-aware summary (when 3 PRs)
+11. Skill generates structured report with root cause traces
+12. Skill executes auto-retest (when all failures are infra flakes)
+13. Skill executes fix-push-rewatch loop (when enabled and fixable failures exist)
+
+The skill receives from the command:
+- Resolved `REPO`, PR numbers, and all parsed flags
+- Operator repo context: `GO_MODULE`, `FRAMEWORK`, `TEST_DIRS`, `HAS_MAKEFILE`, `OPERATOR_CONTEXT_SOURCE` (`local` | `github` | `none`)
+  - When `local`: full repo access for fix loop (can run `make`, edit files, push)
+  - When `github`: read-only context (framework, module path, test dirs) for analysis and reporting; fix loop requires local clone
+  - When `none`: skill falls back to framework-agnostic analysis
+- PR change context: `PR_CHANGED_FILES` (list of file paths changed by each PR) and change type counts (`api`, `controller`, `test`, `crd`, `rbac`)
+  - Used in failure classification to correlate errors with changed files
+  - Used in fix loop to target fixes at files the PR actually touches
+  - Used in stage-aware summary to validate PR staging (e.g., PR #1 should be API-only)
+
+---
+
+*This skill is part of the OAPE AI E2E Feature Development toolkit.*