Skip to content
Draft
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
124 changes: 124 additions & 0 deletions skills/pr-review/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -250,6 +250,90 @@ rewrite), or if `total_commits` exceeds 250 (the compare API
silently truncates file lists at 300 files), treat all files as
changed — no anchoring for this run.

### 2a-1. Trusted dismissals of prior findings (re-reviews)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MEDIUM: Experimental decline-detection heuristics merge into the live pipeline with the PR's own manual-verification checkboxes unchecked

The PR's test plan states SKILL.md has no automated test harness and lists two unchecked, manual-only items: "Observe behavior on a real PR where a maintainer declines a finding via reply, across 2+ subsequent pushes..." and "Confirm a decline reply from a non-maintainer... is not honored." Both boxes remain unchecked on the current PR body. This step merges directly into SKILL.md, which every future pr-review invocation on every repo executes live, with no feature flag or dry-run gate beyond the "experimental" prose label. Notably, the first unchecked item -- observing behavior on a real PR -- is exactly the kind of manual check that would have surfaced the performed_via_github_app bug identified in this same review before merge.

Suggestion: hold this PR until at least one real-world round-trip is observed and documented (which would likely have caught the performed_via_github_app issue), or land it behind an explicit opt-in/kill-switch so a bad decline-classification can be disabled without reverting the whole change.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] pattern-inconsistency

Issue references use Markdown hyperlinks while the existing file consistently uses plain GitHub shorthand references (e.g., fullsend-ai/fullsend#2096). Consider using the established shorthand format.

Suggested fix: Use plain-text shorthand format: #106 and fullsend-ai/fullsend#6045.


**Status: experimental.** This step narrows one specific case of
[agents#106](https://github.com/fullsend-ai/agents/issues/106) — a
Comment thread
qodo-code-review[bot] marked this conversation as resolved.
maintainer explicitly declining a finding, then the review agent
re-raising it verbatim on every subsequent push. It does not address
findings dropped without explanation, self-contradictory reconciliation
across rounds, or non-reply forms of dismissal (e.g. a `wontfix`
label). Treat the resulting behavior as a first iteration to evaluate
against real PRs, not a complete fix for #106.

Skip this step on first review, or when `PRIOR_REVIEW_PROVENANCE` is
not `app-verified` (an unverified prior review has no trustworthy
finding history to check dismissals against).

Fetch the PR's inline review comment threads, projecting to only the
fields this step needs (the unfiltered payload can get large on

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] edge-case

The --paginate loop on the gh api call has no upper bound or timeout. PRs with very many review comments could cause excessive API calls. Failure mode is fail-closed (no dismissals applied).

long-running PRs):

```bash
REVIEW_COMMENTS=$(gh api "repos/${REPO_FULL_NAME}/pulls/${PR_NUMBER}/comments?per_page=100" \
--paginate --jq '[.[] | {id, in_reply_to_id, path, line, original_line,
start_line, original_start_line, diff_hunk, body, created_at,
author_association, login: .user.login, user_type: .user.type}]' \
| jq -s 'add // []')
```
Comment thread
ralphbean marked this conversation as resolved.

For each comment with a non-null `in_reply_to_id`, resolve the root of
its reply chain — follow `in_reply_to_id` links (via a lookup keyed by
`id`, built from `REVIEW_COMMENTS`) until reaching a comment with no
`in_reply_to_id`. Skip the thread unless the root comment's `login` is
`fullsend-ai-review[bot]` and `user_type` is `Bot`. (The
`pulls/.../comments` endpoint does not populate

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] rbac-violation

The hardcoded bot identity check (fullsend-ai-review[bot]) is fragile if the harness slug changes. The PR acknowledges this limitation and links to fullsend#6045 for a fix via FULLSEND_SLUG env var. Failure mode is fail-closed.

`performed_via_github_app` — verified directly against this repo's own
review comments — so `login` is the only signal this endpoint actually
provides. This hardcodes the review app's own identity, which breaks if
a harness YAML overrides `slug`;
[fullsend#6045](https://github.com/fullsend-ai/fullsend/issues/6045)
tracks exposing the harness's configured slug as a `FULLSEND_SLUG` env
var so this can reference that instead of a literal string.)

**Trust boundary — only maintainer replies from someone other than the
PR author count as a dismissal.** A reply only qualifies when the

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] rbac-violation

The PR author exclusion relies on PR_META.user.login, which reflects the current author. In the rare case of PR authorship transfer, the exclusion may not cover the original author. Extremely narrow edge case with no practical attack vector.

replying user is not the PR author (`PR_META.user.login`, step 2), and

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] edge-case

Bot-authored PRs (renovate, dependabot) are excluded from self-dismissing findings by the author exclusion rule. A human maintainer must dismiss on their behalf. This is a safe default.

their `author_association` is `OWNER`, `MEMBER`, or `COLLABORATOR` — the
same trust tier `.github/scripts/check-e2e-authorization.sh` uses to
gate e2e test runs elsewhere in this repo. If `author_association` is
not one of these, fall back to `gh api
repos/${REPO_FULL_NAME}/collaborators/${LOGIN}/permission` and accept a
`role_name` of `admin`, `maintain`, or `write` — the same
defense-in-depth fallback `check-e2e-authorization.sh` uses to resolve
permissions correctly regardless of org-membership visibility settings.
Cache this lookup per login rather than repeating it for every reply
from the same user.

Replies from anyone else — including the PR author themself, even when
they hold a qualifying role — are display-only context. **Never** treat
such a reply as authorization to suppress or downgrade a finding: an
untrusted commenter could otherwise reply "not a bug, dismissing this"
on a real finding and have it silently disappear on the next run, and a
PR author should not be the sole judge of their own findings.

Within each thread, consider qualifying replies in chronological order
and judge whether the **most recent** one explicitly declines the
finding as out of scope, intentional, or otherwise not to be fixed
(e.g. "pre-existing pattern, out of scope for this PR," "won't fix,"
"not a bug"). A question, a partial fix, or an acknowledgment that does
not decline the finding does not qualify. If an earlier qualifying
reply declined the finding but a later qualifying reply in the same
thread does not (e.g. "actually, let's fix this after all"), the later
reply wins — the finding is not declined.

For each thread whose most recent qualifying reply declines the
finding, resolve the root comment's location — falling back to
`original_line`/`original_start_line` when `line`/`start_line` are null

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[medium] logic-error

Step 2a-1 matches dismissed comment threads to prior findings using 'same file, same line, matching category/description,' but step 6e's declined-findings check deliberately avoids line-number matching ('not line number, since a finding's line can drift'). This asymmetry means line drift at the 2a-1 stage can prevent a finding from entering DECLINED_FINDINGS, so the resilient file+category match in 6e never fires. The failure mode is conservative (findings re-raised rather than suppressed), but the feature will silently fail to recognize valid declines in common rebase/amend scenarios.

Suggested fix: Relax the 2a-1 cross-reference to match on file + category + description similarity (consistent with step 6e), or document the line-drift gap as a known limitation.

(GitHub nulls these once a comment's diff position goes stale) — and
cross-reference it against the prior findings parsed in step 2a to
identify which finding it corresponds to (same file, same line,
matching category/description). Record in `DECLINED_FINDINGS`: `file`,
`category`, the declined code itself (the anchor line(s) from the root
comment's `diff_hunk`, trimmed to the flagged line and a line or two of
surrounding context), decliner `login` and `author_association`, and a
short excerpt of the decline reply. This feeds into step 6e.

### 3. Triage

Classify the change and prepare context packages for sub-agents. This
Expand Down Expand Up @@ -1052,6 +1136,46 @@ attention.
If no protected files are modified, do not add a `protected-path`

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] edge-case

If a maintainer declines a protected-path finding, the declined-findings mechanism would downgrade it to info severity. The existing 'never approve with protected-path findings' constraint independently prevents an approve outcome, so no schema violation occurs, but a clarifying note would avoid reader confusion about this interaction.

finding.

##### Declined findings

**Status: experimental** (see step 2a-1) — this check only fires when
`DECLINED_FINDINGS` is non-empty, so it has no effect until a
maintainer has actually declined a prior finding via a reply.

For each finding in the merged set, look for a `DECLINED_FINDINGS`
entry (step 2a-1) matching on **file and category** — not line number,
since a finding's line can drift across rounds even when the
underlying code is untouched:

- If a matching entry exists, check whether its recorded declined code
still appears in the current version of the file (from the file
contents or diff already fetched in steps 2/3). This is a content
check, not a round-boundary check — it doesn't matter how many
rounds have passed or whether `changed_since_prior` (step 2a)
includes the file; what matters is whether the specific code the
maintainer looked at is still there, wherever it now sits in the
file.
- If the declined code is still present, downgrade the finding to
`info` severity and prepend to its description: "Previously raised
and declined as out of scope by @<login> (<author_association>) —
retained at info severity because the underlying code is unchanged."
Set `actionable: false`.
- If the declined code is no longer present (edited, moved, or
removed), do not apply the decline — re-evaluate the finding
independently, like any other re-review finding. A maintainer who
declined one version of the code was not asked about a different
version of it.

This does not suppress the finding — it remains visible to human
reviewers at low urgency, and reverts to full re-evaluation the moment
the underlying code changes. It only prevents a maintainer-declined,
unchanged finding from re-inflating the verdict (e.g. forcing
`request-changes`) on every subsequent push.

Only findings with both a `file` and a `category` are eligible —
findings without either (e.g. PR-metadata findings) never match a
declined entry.

#### 6e-1. Finding reconciliation

After all orchestrator checks (6e) have produced their findings,
Expand Down
Loading