What happened
On PR #6036, the human reviewer (waynesun09) discovered that the new internal/evalmeasure/fitness.go and internal/evalmeasure/types.go files reference OTel semantic convention attribute names (gen_ai.system, gen_ai.request.model, etc.) that were already renamed in upstream OpenTelemetry semconv v1.37.0. Specifically, gen_ai.system was renamed to gen_ai.provider.name. The code shipped with already-stale attribute names. The review agent (fullsend-ai-review[bot], running fullsend-ai/agents@v0, run 32320590280 and 18 subsequent runs) reviewed the same files across 19 runs without flagging this. The review agent's docs-currency sub-agent checks for documentation staleness, and its correctness sub-agent checks for logic errors, but neither checks whether referenced external specifications are current.
What could go better
The review agent has no mechanism to verify that code referencing external standards (OTel semantic conventions, OpenAPI specs, RFC identifiers, W3C trace context versions, etc.) uses the current version of those standards. This is distinct from documentation staleness (handled by docs-currency) and from logic errors (handled by correctness). When code hardcodes attribute names, enum values, or identifiers from an external specification, the review agent should check whether the referenced version is current. Confidence: high — this gap is clearly demonstrated by the PR evidence, and no existing open issue in fullsend-ai/agents or fullsend-ai/fullsend covers upstream specification currency checking (confirmed via 4 targeted searches returning no overlapping results).
Proposed change
Add upstream-spec-currency awareness to the correctness sub-agent in skills/pr-review/sub-agents/correctness.md. When the correctness sub-agent encounters code that references identifiers from well-known external specifications (OTel semantic conventions, HTTP status codes from RFCs, OpenAPI schema versions, W3C trace context fields, etc.), it should note the referenced spec and verify that the identifiers match the current published version. This can be implemented as a prompt addition to the correctness sub-agent instructing it to: (1) identify hardcoded external-standard identifiers in the diff, (2) note which spec version they correspond to, and (3) flag any that are known to be renamed, deprecated, or superseded in more recent versions. The sub-agent should use web search to verify currency when the standard is well-known. Alternatively, this could be a new dedicated sub-agent (spec-currency.md) if the correctness sub-agent's prompt is already at capacity.
Validation criteria
On the next 3 PRs in fullsend-ai/fullsend that add or modify code referencing OTel semantic conventions (or other external standards with versioned attribute names), the review agent should either (a) confirm the referenced attributes match the current spec version, or (b) flag any stale/renamed attributes with a MEDIUM or higher finding. Test with a synthetic PR that uses a known-deprecated OTel attribute name to verify detection.
Generated by retro agent from fullsend-ai/fullsend#6036
What happened
On PR #6036, the human reviewer (waynesun09) discovered that the new
internal/evalmeasure/fitness.goandinternal/evalmeasure/types.gofiles reference OTel semantic convention attribute names (gen_ai.system,gen_ai.request.model, etc.) that were already renamed in upstream OpenTelemetry semconv v1.37.0. Specifically,gen_ai.systemwas renamed togen_ai.provider.name. The code shipped with already-stale attribute names. The review agent (fullsend-ai-review[bot], running fullsend-ai/agents@v0, run 32320590280 and 18 subsequent runs) reviewed the same files across 19 runs without flagging this. The review agent'sdocs-currencysub-agent checks for documentation staleness, and itscorrectnesssub-agent checks for logic errors, but neither checks whether referenced external specifications are current.What could go better
The review agent has no mechanism to verify that code referencing external standards (OTel semantic conventions, OpenAPI specs, RFC identifiers, W3C trace context versions, etc.) uses the current version of those standards. This is distinct from documentation staleness (handled by
docs-currency) and from logic errors (handled bycorrectness). When code hardcodes attribute names, enum values, or identifiers from an external specification, the review agent should check whether the referenced version is current. Confidence: high — this gap is clearly demonstrated by the PR evidence, and no existing open issue in fullsend-ai/agents or fullsend-ai/fullsend covers upstream specification currency checking (confirmed via 4 targeted searches returning no overlapping results).Proposed change
Add upstream-spec-currency awareness to the
correctnesssub-agent inskills/pr-review/sub-agents/correctness.md. When the correctness sub-agent encounters code that references identifiers from well-known external specifications (OTel semantic conventions, HTTP status codes from RFCs, OpenAPI schema versions, W3C trace context fields, etc.), it should note the referenced spec and verify that the identifiers match the current published version. This can be implemented as a prompt addition to the correctness sub-agent instructing it to: (1) identify hardcoded external-standard identifiers in the diff, (2) note which spec version they correspond to, and (3) flag any that are known to be renamed, deprecated, or superseded in more recent versions. The sub-agent should use web search to verify currency when the standard is well-known. Alternatively, this could be a new dedicated sub-agent (spec-currency.md) if the correctness sub-agent's prompt is already at capacity.Validation criteria
On the next 3 PRs in fullsend-ai/fullsend that add or modify code referencing OTel semantic conventions (or other external standards with versioned attribute names), the review agent should either (a) confirm the referenced attributes match the current spec version, or (b) flag any stale/renamed attributes with a MEDIUM or higher finding. Test with a synthetic PR that uses a known-deprecated OTel attribute name to verify detection.
Generated by retro agent from fullsend-ai/fullsend#6036