What happened
On PR #855, the fix agent was addressing a review finding about a stale policy reference in agents/triage.md. In fix iteration 2 (commit fc0d2a8), the fix agent updated the reference from policies/github/triage.yaml to policies/base.yaml and wrote a comment asserting that policies/base.yaml contains a binary allowlist that excludes curl. This was factually incorrect — policies/base.yaml has no binaries section. The curl exclusion works by omission from profile binary allowlists (e.g., profiles/fullsend-github-ro.yaml). The review agent caught this error, noting that the comment "attributes a 'binary allowlist' to policies/base.yaml, but base.yaml contains no binaries section." This required a third fix iteration (commit a2e668e) to correct the comment to reference the profile mechanism.
What could go better
The fix agent wrote a confident factual claim about what policies/base.yaml contains without reading the file. Had it opened policies/base.yaml and observed the absence of a binaries section, it would have searched for where binary allowlists actually live (in profile files like profiles/fullsend-github-ro.yaml) and written the correct comment on the first attempt. This would have eliminated one full review-fix cycle (~35 minutes and 2 agent runs).
This is a specific instance of a general pattern: when the fix agent writes or modifies comments, documentation, or code that makes claims about how other files in the repo work, it should verify those claims by reading the referenced files. Existing issues cover adjacent patterns — #805 addresses factual claims in review-suggested remediation text, #826 addresses fabricated rationale, and #387 addresses the code agent verifying documentation claims — but none cover the fix agent independently generating incorrect domain-specific assertions about repo architecture.
Confidence: High that the root cause is the fix agent not reading the referenced file before asserting its contents. Medium confidence that guidance in the fix agent definition would change the behavior — the agent may have hallucinated the file's contents from training data rather than from a deliberate decision to skip verification.
Proposed change
Add guidance to the fix agent definition (agents/fix.md) instructing the fix agent to verify domain-specific claims by reading source files before committing. Specifically:
-
In the fix agent's instructions, add a rule like: "When writing or modifying comments, documentation, or code that asserts what another file contains or how a mechanism works, read the referenced file(s) first to verify the claim. Do not rely on assumptions about file contents."
-
This should be positioned near existing guidance about code quality and accuracy (if any), as a verification step before committing.
The change is narrowly scoped to the fix agent because:
Validation criteria
Over the next 10 fix agent runs on fullsend-ai/agents that involve modifying comments or documentation referencing other files:
- The fix agent should not produce factual errors about file contents that are catchable by reading the referenced file.
- Review agents should not need to flag incorrect domain-mechanism claims in fix agent commits.
- The average number of fix iterations for documentation/comment changes should decrease (baseline from this PR: 3 iterations for a doc-fix PR).
Generated by retro agent from #855
What happened
On PR #855, the fix agent was addressing a review finding about a stale policy reference in
agents/triage.md. In fix iteration 2 (commitfc0d2a8), the fix agent updated the reference frompolicies/github/triage.yamltopolicies/base.yamland wrote a comment asserting thatpolicies/base.yamlcontains a binary allowlist that excludes curl. This was factually incorrect —policies/base.yamlhas no binaries section. The curl exclusion works by omission from profile binary allowlists (e.g.,profiles/fullsend-github-ro.yaml). The review agent caught this error, noting that the comment "attributes a 'binary allowlist' to policies/base.yaml, but base.yaml contains no binaries section." This required a third fix iteration (commita2e668e) to correct the comment to reference the profile mechanism.What could go better
The fix agent wrote a confident factual claim about what
policies/base.yamlcontains without reading the file. Had it openedpolicies/base.yamland observed the absence of abinariessection, it would have searched for where binary allowlists actually live (in profile files likeprofiles/fullsend-github-ro.yaml) and written the correct comment on the first attempt. This would have eliminated one full review-fix cycle (~35 minutes and 2 agent runs).This is a specific instance of a general pattern: when the fix agent writes or modifies comments, documentation, or code that makes claims about how other files in the repo work, it should verify those claims by reading the referenced files. Existing issues cover adjacent patterns — #805 addresses factual claims in review-suggested remediation text, #826 addresses fabricated rationale, and #387 addresses the code agent verifying documentation claims — but none cover the fix agent independently generating incorrect domain-specific assertions about repo architecture.
Confidence: High that the root cause is the fix agent not reading the referenced file before asserting its contents. Medium confidence that guidance in the fix agent definition would change the behavior — the agent may have hallucinated the file's contents from training data rather than from a deliberate decision to skip verification.
Proposed change
Add guidance to the fix agent definition (
agents/fix.md) instructing the fix agent to verify domain-specific claims by reading source files before committing. Specifically:In the fix agent's instructions, add a rule like: "When writing or modifying comments, documentation, or code that asserts what another file contains or how a mechanism works, read the referenced file(s) first to verify the claim. Do not rely on assumptions about file contents."
This should be positioned near existing guidance about code quality and accuracy (if any), as a verification step before committing.
The change is narrowly scoped to the fix agent because:
Validation criteria
Over the next 10 fix agent runs on fullsend-ai/agents that involve modifying comments or documentation referencing other files:
Generated by retro agent from #855