What happened
PR #3338 removed a test.skip guard from the orchestrator E2E spec that was added by PR #3073 as a fix for issue #3068 (orchestrator backend crash). The review agent approved within 3 minutes, correctly noting the clean removal and the safe failure mode if the bug persists. However, it did not cross-reference the related open issue #3083 — the same crash in the bulk-import-orchestrator workspace — which also carries a test.skip and may now be ready for re-enablement.
What could go better
When a PR removes a test.skip that was added for a product_bug classification, the review agent could search for sibling issues filed for the same root cause across other workspaces. In this case, the skip message contained specific error details (TypeError in BackendInitializer.cjs.js:150) that could be used to find issue #3083. Surfacing stale related issues during review would help maintainers clean up test.skip guards and close resolved product_bug issues faster.
Confidence: Medium. The review agent's primary job is code quality assessment, and it performed that well. Cross-referencing related issues is an incremental improvement that adds a manual checkpoint. Existing proposal #3302 (auto-close product_bug issues after passing nightlies) would address the root cause more systematically — this proposal is complementary, not a substitute.
Proposed change
Add a subsection to the AGENTS.md 'E2E Nightly Fix Conventions' section titled 'Reviewing test.skip removal PRs'. The guidance should instruct reviewers that when a PR removes a test.skip added for a product_bug classification:
- Search open issues for the same root cause string from the skip message to identify related skips in other workspaces
- Note any related open issues in the review comment so maintainers can evaluate them for re-enablement
- Verify that the PR's test results confirm the underlying bug is resolved (e.g., E2E tests pass in the affected mode)
This is a lightweight addition (~10 lines) to the existing E2E conventions section, consistent with the pattern of other AGENTS.md review guidance issues (#2969, #3017, #3177).
Validation criteria
On the next PR that removes a test.skip for a product_bug classification, the review agent should: (1) identify and link related open issues with the same root cause, and (2) note whether those related issues may also be ready for re-enablement. Measure over the next 3 such PRs within 90 days.
Generated by retro agent from #3338
What happened
PR #3338 removed a
test.skipguard from the orchestrator E2E spec that was added by PR #3073 as a fix for issue #3068 (orchestrator backend crash). The review agent approved within 3 minutes, correctly noting the clean removal and the safe failure mode if the bug persists. However, it did not cross-reference the related open issue #3083 — the same crash in the bulk-import-orchestrator workspace — which also carries atest.skipand may now be ready for re-enablement.What could go better
When a PR removes a
test.skipthat was added for aproduct_bugclassification, the review agent could search for sibling issues filed for the same root cause across other workspaces. In this case, the skip message contained specific error details (TypeError in BackendInitializer.cjs.js:150) that could be used to find issue #3083. Surfacing stale related issues during review would help maintainers clean up test.skip guards and close resolved product_bug issues faster.Confidence: Medium. The review agent's primary job is code quality assessment, and it performed that well. Cross-referencing related issues is an incremental improvement that adds a manual checkpoint. Existing proposal #3302 (auto-close product_bug issues after passing nightlies) would address the root cause more systematically — this proposal is complementary, not a substitute.
Proposed change
Add a subsection to the AGENTS.md 'E2E Nightly Fix Conventions' section titled 'Reviewing test.skip removal PRs'. The guidance should instruct reviewers that when a PR removes a
test.skipadded for aproduct_bugclassification:This is a lightweight addition (~10 lines) to the existing E2E conventions section, consistent with the pattern of other AGENTS.md review guidance issues (#2969, #3017, #3177).
Validation criteria
On the next PR that removes a
test.skipfor a product_bug classification, the review agent should: (1) identify and link related open issues with the same root cause, and (2) note whether those related issues may also be ready for re-enablement. Measure over the next 3 such PRs within 90 days.Generated by retro agent from #3338