Review agent should not produce unsubstantiated verification claims

[fullsend-ai-retro]

What happened

During the 13+ review iterations on PR #211, the review bot produced freeform text claiming "zero references to deleted files remain" and "GH_TOKEN delivery chain verified" at commits where both were provably false. The human reviewer (waynesun09) demonstrated via CI artifact analysis that stale host_files references to deleted env files persisted, and that GH_TOKEN: "${GH_TOKEN}" was re-added to forge.github.env.sandbox in commit 8a0ad3c. The pr-review skill's prescribed output format (SKILL.md) defines a structured findings format but the agent generated additional verification claims outside this format. These false verifications are more dangerous than missing findings — they create false confidence that could lead to premature merges.

What could go better

The review agent generated text asserting it had verified runtime properties (reference integrity, credential flow) that it could not actually verify from static diff analysis alone. The prescribed output format does not include a "Verified" section, yet the agent produced one. The challenger sub-agent is designed to remove false positive findings but does not challenge the agent's own verification claims. Confidence is high: the human reviewer explicitly demonstrated both claims were false with evidence from CI artifacts. The root cause is that the agent's instructions do not prohibit extrapolating from spot-checks to blanket verification claims.

Proposed change

Add an explicit prohibition in skills/pr-review/SKILL.md (output format section) and/or agents/review.md against unsubstantiated verification claims. Specifically:

1. In the SKILL.md output format instructions, add: "Do not include freeform verification sections or claim to have verified properties beyond what the diff and source files directly show. When a prior finding is no longer present in the reviewed diff, state 'finding not observed in current diff' rather than 'verified resolved.' Never claim exhaustive verification of reference integrity, credential flows, or runtime behavior — these require CI/runtime validation the review agent cannot perform."

2. In sub-agents/challenger.md, add an instruction to challenge any verification claims in the aggregated output, not just findings: "If the aggregated output contains claims of verification (e.g., 'all references verified', 'delivery chain confirmed'), challenge whether the agent actually performed exhaustive checks to support that claim. Remove unsubstantiated verification text."

Validation criteria

Over the next 10 review runs on fullsend-ai/agents, the review body should not contain blanket verification claims (phrases like 'verified: ✅', 'zero X remain', 'delivery chain verified') unless they cite the specific files/lines checked. When prior findings are no longer flagged, the body should use qualified language ('not observed in current diff') rather than absolute claims ('verified resolved').

---

Generated by retro agent from #211

[fullsend-ai-triage]

🤖 Finished Triage · ✅ Success · Started 12:30 PM UTC · Completed 12:34 PM UTC

Commit: 7aa8587 · View workflow run →

[fullsend-ai-triage]

Triage assessment

Severity: high — false verification claims are actively worse than missing findings because they create false confidence and risk premature merges.

Related issues

• Review agent should verify its own technical assertions against source code before posting findings #420 (open): "Review agent should verify its own technical assertions against source code before posting findings" — addresses a related but distinct problem (false runtime assertions within structured findings). This issue covers freeform verification outside the findings format. Both should be addressed independently.
• Review agent should treat defensive-code removal with library-delegation claims as unverifiable #639 (open): "Review agent should treat defensive-code removal with library-delegation claims as unverifiable" — tangentially related epistemological concern.

Implementation notes

The two proposed changes are well-scoped:

1. skills/pr-review/SKILL.md — Add to the output format section: prohibit freeform verification sections; require qualified language for resolved findings.
2. skills/pr-review/sub-agents/challenger.md — Expand challenger scope beyond findings to also challenge verification claims in the aggregated output.

Note that PR #753 also modifies the pr-review SKILL.md (tightening weak language) — coordinate to avoid merge conflicts if both land concurrently.

Proposed test case

Scenario: Review agent does not produce blanket verification claims
  Given a PR diff where a prior review flagged missing references
  When the review agent produces its review body
  Then the body should NOT contain phrases like:
    - "verified: ✅"
    - "zero X remain"
    - "delivery chain verified"
    - "verified resolved"
  And if a prior finding is no longer present, the body should say:
    - "not observed in current diff"
  And the challenger sub-agent should flag and remove any
    unsubstantiated verification text from aggregated output

---

Labels: This is a review-agent issue targeting the pr-review skill's output format and challenger sub-agent behavior. It describes a documentation/instruction gap (not a code defect) that should be fixed by updating agent instructions.

---

Next steps:

• /fs-code — agent creates a PR to implement this issue
• /fs-code <your instruction> — agent implements with your specific guidance

[fullsend-ai-coder]

🤖 Finished Code · ✅ Success · Started 12:35 PM UTC · Completed 12:41 PM UTC

Commit: 7aa8587 · View workflow run →