feat(retro): add flapping detection to retro analysis skill - #834
feat(retro): add flapping detection to retro analysis skill#834Benkapner wants to merge 1 commit into
Conversation
Teach the retro agent to detect fix-break oscillation patterns during post-workflow analysis. Covers three flapping signals: file oscillation (reversing diffs across runs), test result flipping (pass/fail/pass correlated to agent changes), and review-fix cycle count (same or alternating findings without convergence). Uses the event_payload JSON in dispatch-repo run logs to correlate runs to PR commits directly, avoiding timestamp-based approximation. Supersedes fullsend-ai#540, which accumulated 7 review rounds and diverged from this file's conventions. This version keeps the section at the same altitude as the rest of the file. Closes fullsend-ai/fullsend#5512 Signed-off-by: Benjamin Kapner <bkapner@redhat.com>
Functional tests did not runFunctional tests run automatically for org/repo members and collaborators on pull requests. For other contributors, a maintainer must add the |
PR Summary by QodoAdd flapping detection guidance to retro-analysis skill
AI Description
Diagram
High-Level Assessment
Files changed (1)
|
Code Review by Qodo
1. Protected skills/ file modified
|
![qodo-code-review[bot]](https://avatars.githubusercontent.com/in/484649?s=60&v=4){kind=small}
| ## Flapping detection | ||
|
|
||
| Check whether the workflow exhibits fix-break oscillation. Flapping wastes agent cycles and often indicates a deeper problem (conflicting instructions, flaky tests, or an approach the agent cannot converge on). |
There was a problem hiding this comment.
1. Protected skills/ file modified 📜 Skill insight § Compliance
This PR modifies skills/retro-analysis/SKILL.md, which is a protected governance/infrastructure path requiring explicit human review and must not be auto-approved. Ensure appropriate reviewers/CODEOWNERS sign off before merge.
Agent Prompt
## Issue description
The PR changes a protected path (`skills/`), which must not be auto-approved and requires explicit human review.
## Issue Context
Protected governance/infrastructure paths require elevated scrutiny. This PR updates the retro-analysis skill content under `skills/`.
## Fix Focus Areas
- skills/retro-analysis/SKILL.md[124-126]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
|
|
||
| Dispatch a subagent to identify code/fix/review workflow runs for the PR and collect the data needed for pattern detection: | ||
|
|
||
| - **Flapping data collector:** "Find all code, fix, and review workflow runs related to PR #<PR_NUMBER> in `<DISPATCH_REPO>`. Each run's log contains an `event_payload` JSON line with `pull_request.head.sha` and `pull_request.number`; parse it to correlate runs to PR commits and to confirm the run belongs to this PR. For each matched run, fetch the commit's changed files and CI check-run results from `<REPO>`. Also fetch the PR's review comments/findings (`--paginate`) so finding content can be compared across review cycles." |
There was a problem hiding this comment.
2. Unresolved repo placeholders 🐞 Bug ≡ Correctness
The new flapping data collector prompt uses literal <DISPATCH_REPO> and <REPO> tokens even though this skill defines $DISPATCH_REPO and uses $REPO_FULL_NAME in its recipes. If these placeholders aren’t substituted, the subagent can run gh commands against an invalid/wrong repo and the flapping analysis will fail or collect the wrong data.
Agent Prompt
### Issue description
The flapping data collector prompt uses `<DISPATCH_REPO>` and `<REPO>` placeholders that are not established elsewhere in the skill, despite the skill defining `$DISPATCH_REPO` and consistently using `$REPO_FULL_NAME` for repo-qualified `gh` calls.
### Issue Context
Within `skills/retro-analysis/SKILL.md`, the Setup section defines `DISPATCH_REPO` and subsequent commands use `$DISPATCH_REPO` / `$REPO_FULL_NAME`. The newly added flapping collector prompt should follow the same convention (or explicitly define how placeholders are substituted).
### Fix Focus Areas
- skills/retro-analysis/SKILL.md[124-136]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
|
|
||
| Dispatch a subagent to identify code/fix/review workflow runs for the PR and collect the data needed for pattern detection: | ||
|
|
||
| - **Flapping data collector:** "Find all code, fix, and review workflow runs related to PR #<PR_NUMBER> in `<DISPATCH_REPO>`. Each run's log contains an `event_payload` JSON line with `pull_request.head.sha` and `pull_request.number`; parse it to correlate runs to PR commits and to confirm the run belongs to this PR. For each matched run, fetch the commit's changed files and CI check-run results from `<REPO>`. Also fetch the PR's review comments/findings (`--paginate`) so finding content can be compared across review cycles." |
There was a problem hiding this comment.
3. Assumed event_payload log line 🐞 Bug ☼ Reliability
The flapping data collector prompt unconditionally claims each dispatch-repo run log contains an event_payload JSON line with pull_request.head.sha and pull_request.number. Repo documentation for locating dispatch runs currently relies on timestamp/headBranch correlation and does not establish this log line as a guaranteed interface, so the new guidance can break flapping detection if the log format differs or the line is absent.
Agent Prompt
### Issue description
The new flapping collector instructions depend on a specific `event_payload` JSON log line for correlating runs to PR commits. This is not documented elsewhere as a guaranteed log output, and existing run-finding guidance uses timestamp/headBranch matching.
### Issue Context
`skills/finding-agent-runs/SKILL.md` demonstrates the established approach: find shim runs in the source repo and match dispatch-repo runs by timestamp/headBranch. `scripts/pre-code.sh` and `scripts/pre-fix.sh` mention `event_payload` only as an input-validation concern, not as a logged artifact.
### Fix Focus Areas
- skills/retro-analysis/SKILL.md[124-137]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
waynesun09
left a comment
waynesun09
left a comment
There was a problem hiding this comment.
Automated review sweep: 5 findings on the flapping detection addition (1 critical, 4 medium).
|
|
||
| ### Patterns to detect | ||
|
|
||
| 1. **File oscillation:** the same file was changed in two or more consecutive runs, and the changes reverse each other (lines added in run N were removed in run N+1, or vice versa). |
There was a problem hiding this comment.
CRITICAL: Pattern 1 (file oscillation) fires on a single reversal, contradicting the skill's own "When NOT to flag" rule
Pattern 1's definition (line 140: "the same file was changed in two or more consecutive runs, and the changes reverse each other ... lines added in run N were removed in run N+1") triggers on a single N/N+1 reversal — exactly two runs. But the "When NOT to flag" section added in the same diff (lines 155-159) states "A single rework cycle (review requested changes, fix addressed them, review approved) is normal" and "Only flag when you see the same changes being applied and reversed repeatedly." These two sections of the same PR contradict each other: following Pattern 1 literally will flag the default happy-path review-fix cycle (code adds X, review flags X, fix removes X) as flapping, which is precisely the false-positive the exclusion section exists to prevent.
This is compounded by "consecutive runs" being ambiguous given the real workflow sequence is code → review → fix → review → fix, where review runs typically touch no files — if "consecutive" means consecutive workflow runs, oscillation almost never fires; if it means consecutive file-changing runs, it collapses back into the single-reversal false positive. It's also the only one of the three patterns with a 2-run threshold: Pattern 2 requires pass-fail-pass (3 runs) and Pattern 3 requires "more than 2" cycles (3+), so Pattern 1 is inconsistent with its siblings as well as with the exclusion list.
Suggestion: align Pattern 1 with the "repeated reversals" bar used everywhere else in the section — require at least two reversals (e.g. A→B→A across three file-changing runs) rather than a single undo, and explicitly define "consecutive" as consecutive file-changing (code/fix) runs, with review runs used only to correlate finding text rather than for the file-diff comparison.
|
|
||
| ### Applicability | ||
|
|
||
| Flapping detection applies to PR-based workflows with code/fix cycles. If `$ORIGINATING_URL` is an issue URL, check whether a PR is linked (`gh issue view "$ORIGINATING_URL" --json closedByPullRequestsReferences`) before skipping. If no linked PR exists, skip flapping detection for this retro. |
There was a problem hiding this comment.
MEDIUM: <PR_NUMBER> placeholder used in the data-gathering prompt is never derived
The Applicability section (this line) only covers the case where $ORIGINATING_URL is an issue URL ("check whether a PR is linked ... before skipping"); it says nothing about the common case where $ORIGINATING_URL is already a PR URL, and in neither branch does it bind a value to the <PR_NUMBER> placeholder used two lines later in the data-gathering prompt ("Find all code, fix, and review workflow runs related to PR #<PR_NUMBER>", line 136). No earlier section in SKILL.md defines $ORIGINATING_URL, PR_NUMBER, or any variable a subagent could substitute here (the Setup section only defines $REPO_FULL_NAME/$DISPATCH_REPO). A subagent following this literally has no stated source for <PR_NUMBER> in either branch.
Suggestion: add one line per branch — if $ORIGINATING_URL is a PR URL, extract its number directly; if it's an issue URL, use the linked PR's number from closedByPullRequestsReferences — and reference that resolved value explicitly when introducing the data-gathering prompt.
|
|
||
| Dispatch a subagent to identify code/fix/review workflow runs for the PR and collect the data needed for pattern detection: | ||
|
|
||
| - **Flapping data collector:** "Find all code, fix, and review workflow runs related to PR #<PR_NUMBER> in `<DISPATCH_REPO>`. Each run's log contains an `event_payload` JSON line with `pull_request.head.sha` and `pull_request.number`; parse it to correlate runs to PR commits and to confirm the run belongs to this PR. For each matched run, fetch the commit's changed files and CI check-run results from `<REPO>`. Also fetch the PR's review comments/findings (`--paginate`) so finding content can be compared across review cycles." |
There was a problem hiding this comment.
MEDIUM: <REPO> in the data-gathering prompt is not bound to the linked PR's repo in the cross-repo issue case
This PR's own linked issue demonstrates the gap: the issue lives in one repo and the linked PR (this one) lives in a different repo (closedByPullRequestsReferences on the issue resolves to a PR in a different repo than the issue itself). The Applicability check resolves a linked PR from an issue URL but never states which repo <REPO> (used here: "fetch the commit's changed files and CI check-run results from <REPO>") should resolve to when the linked PR lives in a different repo than $REPO_FULL_NAME.
Note this is distinct from the existing bot review comment on this line (which flags that <DISPATCH_REPO>/<REPO> are literal unsubstituted tokens vs. the skill's $DISPATCH_REPO/$REPO_FULL_NAME convention) — even if that comment's suggested fix of substituting $REPO_FULL_NAME is applied, it would resolve to the wrong repo in exactly this cross-repo scenario, since the actual PR/CI data lives in the linked PR's own repo, not the issue's repo.
Suggestion: explicitly bind <REPO> to the repository field returned by the linked PR (from closedByPullRequestsReferences) when flapping detection is entered via the issue-URL branch, noting it may differ from $REPO_FULL_NAME.
| ### Patterns to detect | ||
|
|
||
| 1. **File oscillation:** the same file was changed in two or more consecutive runs, and the changes reverse each other (lines added in run N were removed in run N+1, or vice versa). | ||
| 2. **Test result flipping:** a test that passed after run N fails after run N+1, then passes again after run N+2, and the flapping test covers a file the agent modified in the same run. Tests that flip independently of agent changes may be pre-existing flaky tests, not agent-caused oscillation. |
There was a problem hiding this comment.
MEDIUM: Pattern 2 (test result flipping) needs per-test/file-coverage data the collector never fetches
Pattern 2 requires identifying "a test that passed after run N fails after run N+1, then passes again after run N+2, and the flapping test covers a file the agent modified in the same run" — i.e., both a specific failing test and a test-to-file coverage mapping. The only CI data the collector prompt (line 136) fetches is "CI check-run results," which GitHub's Checks API returns at job/suite granularity (e.g. a single "unit-tests" check), not per-test, and carries no file-coverage mapping. No heuristic is given for how a subagent would derive individual test identity or test-to-file coverage from check-run-level data alone.
Suggestion: either specify a concrete heuristic (e.g. parse per-test names out of CI log output/test-report artifacts, if available) or relax Pattern 2 to what check-run-level data actually supports (e.g. "the same named CI job flips status across 3+ runs while covering the same changed files").
|
|
||
| 1. **File oscillation:** the same file was changed in two or more consecutive runs, and the changes reverse each other (lines added in run N were removed in run N+1, or vice versa). | ||
| 2. **Test result flipping:** a test that passed after run N fails after run N+1, then passes again after run N+2, and the flapping test covers a file the agent modified in the same run. Tests that flip independently of agent changes may be pre-existing flaky tests, not agent-caused oscillation. | ||
| 3. **Cycle count:** more than 2 review-fix cycles on the same PR without convergence (the review keeps raising the same or alternating findings, e.g. a fix for one issue reintroducing a previously resolved one). A single rework cycle where the fix addresses the feedback and the review approves is normal iteration, not flapping. |
There was a problem hiding this comment.
MEDIUM: Hardcoded "more than 2" cycle threshold presents an explicitly unresolved design question as settled
Pattern 3 states as fact that "more than 2 review-fix cycles on the same PR without convergence" is flapping. The design doc this feature implements (docs/problems/flapping-convergence.md in fullsend-ai/fullsend) explicitly says thresholds must vary per repo/task type in its "Thresholds and configuration" section ("A documentation repo might tolerate only 2 review cycles ... A complex backend service might allow 5 cycles ... Default thresholds should be conservative and configurable per repo and per agent role"), and its "Open questions" section lists "What is the right default flapping budget?" as unanswered. This PR hardcodes a fixed, non-configurable "2" in the skill without referencing the doc or noting it as a provisional default.
Suggestion: reference docs/problems/flapping-convergence.md and phrase the threshold as a starting heuristic pending configurable per-repo thresholds, rather than an evidence-based fixed number.
2 participants
Summary
Adds a flapping detection section to the retro-analysis skill, teaching the retro agent to identify fix-break oscillation patterns during post-workflow analysis.
Supersedes #540, which accumulated 7 review rounds and diverged from the file's conventions. This version incorporates the key insight from that review process (the
event_payloadJSON in dispatch-repo logs exposespull_request.head.shaandpull_request.numberdirectly) and keeps the section at the same altitude as the rest of the file.What it adds
Guidance for detecting three flapping patterns:
Design decisions
event_payloadfrom dispatch-repo run logs for run-to-commit correlation (no timestamp approximation)Related
Checklist