Triage agent should verify implementation-level code snippets against actual source before finalizing plan

[fullsend-ai-retro]

What happened

On issue #6188, the triage agent's initial plan (run 31716452078) produced 4 concrete errors in code-level implementation details:

1. Reversed function signature: Plan used mergeBaseIntoChild(&child, &base) but the actual signature is mergeBaseIntoChild(base, child *Harness) — base first, per compose.go:524.
2. Missing required struct fields in test fixtures: Test sketches used bare Harness{Effort: level} which would fail validation on required Agent/Role fields before effort validation was reached.
3. Missing plumbing step: The plan omitted threading Effort through the RunParams struct in internal/runtime/runtime.go and its construction site in internal/cli/run.go.
4. Missing guard pattern: Plan didn't specify the if child.Effort == "" { ... } guard in composition, needed to match the existing Model field pattern.

The human (waynesun09) spent time validating the triage against main, wrote 4 detailed corrections, and re-triggered /fs-triage. The re-triage incorporated all corrections faithfully and the subsequent code agent run produced a clean implementation.

What could go better

The triage agent generated plausible-looking but incorrect code snippets without reading the actual source files it referenced. All 4 errors were verifiable by reading the files the plan already identified:

• Reading compose.go would reveal the mergeBaseIntoChild(base, child *Harness) signature and the if child.Model == "" guard pattern.
• Reading harness.go would reveal Agent and Role are required for validation to pass.
• Reading runtime.go would reveal the RunParams struct that needs the new field threaded through.

This is related to #2251 (validate root-cause hypotheses by tracing code) but covers a distinct failure mode: feature-triage implementation plans containing hallucinated API contracts, rather than bug-triage misdiagnosis. The underlying principle — verify claims against source — is the same, but the fix targets a different triage phase (prescription vs. diagnosis).

Confidence: High that the errors occurred and were avoidable by reading source. Moderate that a verification step would reliably catch this class of error in general.

Proposed change

Add a verification step to the triage agent's planning phase: when the plan includes code-level implementation details (function calls, struct literals, test patterns, inheritance behavior), the agent should read the referenced source files to verify:

1. Function signatures match actual definitions (argument order, types, receiver).
2. Struct literals include all fields required for validation to reach the new code path.
3. All plumbing steps are accounted for — if field A is added to struct X and consumed in function Y, verify the path from X to Y.
4. Guard/inheritance patterns match existing sibling fields (e.g., the Model field pattern).

This change likely belongs in the triage agent definition or skills. The agents repo could not be discovered from the workflow run logs for this PR (no Fetching agent ... from ... pattern found). If the triage agent definition lives outside fullsend-ai/fullsend, this proposal should be re-targeted to the appropriate agents repo. As an alternative or supplement, AGENTS.md in this repo could include triage-specific guidance requiring implementation plan details to be verified against source.

Validation criteria

On the next 5 feature-triage runs in this repo where the plan includes code-level implementation details (function calls, struct patterns), check whether: (1) function signatures match actual source, (2) test fixture patterns include all required fields, (3) no plumbing steps are omitted. Success: 0 of 5 plans contain verifiable factual errors about the codebase's API contracts. Current baseline from this PR: 1 of 1 plans had 4 such errors.

---

Generated by retro agent from #6218

[fullsend-ai-triage]

🤖 Finished Triage · ✅ Success · Started 4:00 PM UTC · Completed 4:04 PM UTC

Commit: 711d6c7 · View workflow run →

[fullsend-ai-triage]

Triage summary

Severity: Medium — causes human rework but the triage→correction→re-triage cycle ultimately produced a correct plan.

Related work:

• Triage agent should validate root-cause hypotheses by tracing code before dispatching #2251 covers the diagnosis-phase equivalent (validating root-cause hypotheses by tracing code). This issue targets the prescription phase (verifying implementation plan details). Both share the same underlying principle but address different triage phases.
• Triage agent should not assert external CLI commands don't exist without verification #1856 addresses a narrower case (asserting CLI commands don't exist without verification).
• The triage agent definition lives in fullsend-ai/agents (agents/triage.md). The most impactful fix would add verification logic there — either in the agent prompt or as a new skill. AGENTS.md guidance in this repo is a lighter-weight supplement.

Proposed test case:

For the next 5 feature-triage runs with code-level details:
1. Compare each function signature in the plan against the actual
   source definition (argument order, types, receiver).
2. Verify struct literals include all fields required for validation
   to reach the new code path.
3. Confirm all plumbing steps are present — if field A is added to
   struct X and consumed in function Y, the path X→Y must be complete.
4. Check guard/inheritance patterns match existing sibling fields.

Pass: 0 of 5 plans contain verifiable API contract errors.
Baseline: 1 of 1 plan had 4 such errors.

---

Labels: This is a feature request targeting the triage agent, filed by the retro agent with concrete evidence. Priority is medium — impacts plan quality but does not block triage from eventually succeeding.

---

Next steps:

• /fs-code — agent creates a PR to implement this issue
• /fs-code <your instruction> — agent implements with your specific guidance