What happened
On PR #722, the eval/measurements/README.md documented that managed jobs fetch stock manifests via an unauthenticated raw.githubusercontent.com URL. The PR body explicitly referenced the companion engine PR fullsend-ai/fullsend#6036, which implements the actual fetch mechanism as authenticated, SHA-pinned GetRef calls. The companion PR includes a test (TestActionYML_EvalMeasureNoFloatingV0Curl) asserting the action.yml does NOT contain raw.githubusercontent.com. The human reviewer verified this by reading the companion PR's code at commit 3d5aefc6 and flagged it as HIGH severity on Aug 17. The review agent ran 8 successful reviews across Aug 10--20 without ever flagging this discrepancy -- its docs-currency and cross-repo-contracts sub-agents did not inspect the companion PR's code to verify the documentation's claims.
What could go better
When a PR body explicitly couples itself to a companion PR (via references like 'Companion engine PR: fullsend-ai/fullsend#6036'), the review agent's sub-agents should fetch and read relevant files from the companion to verify documentation claims. The existing cross-repo-contracts sub-agent has a cross-repo mandate but focuses on API contract breakage, not documentation accuracy. The docs-currency sub-agent checks for staleness against the current diff but doesn't cross-reference companion PRs. This is a high-confidence assessment: the PR body contained an explicit companion reference, the documentation made a specific testable claim about a URL scheme, and the claim directly contradicted the companion's implementation and test suite. Confidence is tempered by the practical challenge of determining which companion files are relevant to which documentation claims -- not all documentation claims are as mechanically verifiable as a URL pattern.
Proposed change
Add guidance to the cross-repo-contracts sub-agent definition in skills/pr-review/sub-agents/ instructing it to: (1) parse the PR body for explicit companion PR references (patterns like 'Companion PR:', 'Depends on:', or inline owner/repo#N references with coupling language), (2) when documentation in the current PR asserts specific implementation behavior from those companions (URL schemes, API mechanisms, fetch strategies, security properties), use gh api or gh pr diff to read the companion's relevant files, and (3) flag any documentation claim that contradicts the companion's actual code as a HIGH finding. The REVIEW_TOKEN already provides cross-repo API access. This extends the scope of existing issue #726 (which covers PR state verification) to include content-level verification of companion PR behavior.
Validation criteria
On the next PR that explicitly references a companion PR and includes documentation asserting specific behavior from that companion, the review agent should produce a finding if the documentation contradicts the companion's code. Measurable over the next 5 cross-repo coupled PRs in this repository.
Generated by retro agent from #722
What happened
On PR #722, the
eval/measurements/README.mddocumented that managed jobs fetch stock manifests via an unauthenticatedraw.githubusercontent.comURL. The PR body explicitly referenced the companion engine PR fullsend-ai/fullsend#6036, which implements the actual fetch mechanism as authenticated, SHA-pinned GetRef calls. The companion PR includes a test (TestActionYML_EvalMeasureNoFloatingV0Curl) asserting the action.yml does NOT containraw.githubusercontent.com. The human reviewer verified this by reading the companion PR's code at commit3d5aefc6and flagged it as HIGH severity on Aug 17. The review agent ran 8 successful reviews across Aug 10--20 without ever flagging this discrepancy -- its docs-currency and cross-repo-contracts sub-agents did not inspect the companion PR's code to verify the documentation's claims.What could go better
When a PR body explicitly couples itself to a companion PR (via references like 'Companion engine PR: fullsend-ai/fullsend#6036'), the review agent's sub-agents should fetch and read relevant files from the companion to verify documentation claims. The existing cross-repo-contracts sub-agent has a cross-repo mandate but focuses on API contract breakage, not documentation accuracy. The docs-currency sub-agent checks for staleness against the current diff but doesn't cross-reference companion PRs. This is a high-confidence assessment: the PR body contained an explicit companion reference, the documentation made a specific testable claim about a URL scheme, and the claim directly contradicted the companion's implementation and test suite. Confidence is tempered by the practical challenge of determining which companion files are relevant to which documentation claims -- not all documentation claims are as mechanically verifiable as a URL pattern.
Proposed change
Add guidance to the cross-repo-contracts sub-agent definition in
skills/pr-review/sub-agents/instructing it to: (1) parse the PR body for explicit companion PR references (patterns like 'Companion PR:', 'Depends on:', or inlineowner/repo#Nreferences with coupling language), (2) when documentation in the current PR asserts specific implementation behavior from those companions (URL schemes, API mechanisms, fetch strategies, security properties), usegh apiorgh pr diffto read the companion's relevant files, and (3) flag any documentation claim that contradicts the companion's actual code as a HIGH finding. TheREVIEW_TOKENalready provides cross-repo API access. This extends the scope of existing issue #726 (which covers PR state verification) to include content-level verification of companion PR behavior.Validation criteria
On the next PR that explicitly references a companion PR and includes documentation asserting specific behavior from that companion, the review agent should produce a finding if the documentation contradicts the companion's code. Measurable over the next 5 cross-repo coupled PRs in this repository.
Generated by retro agent from #722