Auto-close product_bug E2E issues when nightly failures stop reproducing

[fullsend-ai-retro]

What happened

Issue #3069 was correctly classified as product_bug by the triage agent after the scorecard entity-level view showed "Metric data unavailable" in the nightly E2E run. The code agent correctly created PR #3074 with test.skip guards. However, the underlying product bug was fixed upstream independently. The issue and PR sat open for 17 days until a human (imykhno) manually noticed the failure had stopped reproducing and closed both on Aug 18.

What could go better

There is no automated mechanism to detect when a product_bug-classified E2E failure stops reproducing in nightly runs. When the scorecard tests started passing again in subsequent nightlies, the system should have detected this and flagged or auto-closed the issue and its linked PR. Instead, a human had to manually monitor nightly results and close the stale issue/PR. This wastes human attention and leaves unnecessary test.skip PRs open, contributing to review backlog. Confidence: high — this is a clear lifecycle gap. The triage system creates issues when failures appear but has no symmetric mechanism to close them when failures disappear.

Proposed change

Add a post-nightly reconciliation step (either in the e2e-ocp-helm-nightly workflow or as a separate workflow triggered after nightly completion) that cross-references nightly test results with open product_bug E2E issues. Specifically:

1. After the nightly E2E run, collect the set of tests that passed.
2. Query open issues with [fullsend] E2E: prefix or e2e-failure label whose fix_category is product_bug.
3. For each open issue, check if the test(s) it references passed in the current nightly run.
4. If a test passes for N consecutive nightly runs (suggest N=3 to filter transient passes), comment on the issue noting the failure is no longer reproducing and auto-close the issue. If there is a linked agent PR (branch pattern agent/{issue}-*), close that PR as well with a comment explaining the resolution.

The test-to-issue mapping could use the workspace name and test name from the issue title (e.g., "scorecard — entity-level filecheck") matched against the JUnit/Playwright test results from the nightly run artifacts.

Validation criteria

The next time a product_bug E2E issue self-resolves (the underlying failure stops reproducing in nightlies for 3+ consecutive runs), the system should auto-close the issue and any linked agent PRs within 4 days of the failure stopping, without requiring manual human intervention. Measure over 90 days: count product_bug issues closed by automation vs. manually, targeting >50% automation rate.

---

Generated by retro agent from #3074

[fullsend-ai-triage]

🤖 Finished Triage · ✅ Success · Started 10:26 AM UTC · Completed 10:30 AM UTC

Commit: 6697e15 · View workflow run →

[fullsend-ai-triage]

Workflow changes required

⚠ This feature requires creating or modifying GitHub Actions workflow files (.github/workflows/). The code agent cannot modify workflow files under current permissions — manual intervention (human PR or maintainer action) is required.

Implementation considerations

The existing e2e-triage-agent.yaml already parses Playwright results.json from GCSWeb artifacts to extract pass/fail stats. The reconciliation step could reuse this infrastructure to also extract per-test results and cross-reference them with open product_bug issues.

Key design decisions for the implementer:

• Test-to-issue mapping: The [fullsend] E2E: issue title format includes workspace and test description (e.g., "scorecard — entity-level filecheck"). Matching this to Playwright test IDs may require fuzzy matching or storing the exact test path in the issue body at creation time.
• Consecutive-pass tracking: Options include (a) issue comments as state (simplest, but noisy), (b) a state file in the repo or GCS, or (c) querying the last N nightly runs from GCSWeb.
• Linked PR cleanup: Agent PRs follow the branch pattern agent/{issue}-* — the workflow should close these with a comment explaining the resolution.

Related issues

• Add failure alerting for the e2e-triage-agent pipeline #3213 addresses alerting when the triage pipeline itself fails — complementary but distinct from this lifecycle management request.

# Proposed test case (pseudocode)
def test_reconciliation_closes_after_consecutive_passes():
    """Verify product_bug issues auto-close after N consecutive nightly passes."""
    # Setup: open issue with product_bug classification
    issue = create_issue(title="[fullsend] E2E: scorecard — entity-level filecheck")
    
    # Simulate 3 consecutive passes
    for run in range(3):
        results = {"scorecard.entity-level-filecheck": "passed"}
        reconcile(results, open_issues=[issue])
    
    # Assert issue is closed with explanatory comment
    assert issue.state == "closed"
    assert "no longer reproducing" in issue.comments[-1]
    
    # Edge case: reset counter on intermittent failure
    issue2 = create_issue(title="[fullsend] E2E: topology — render check")
    reconcile({"topology.render-check": "passed"}, [issue2])  # pass 1
    reconcile({"topology.render-check": "failed"}, [issue2])  # fail resets
    reconcile({"topology.render-check": "passed"}, [issue2])  # pass 1 again
    assert issue2.state == "open"  # only 1 consecutive pass

---

Labels: This is a feature request for new CI/workflow automation (enhancement). It touches the E2E triage pipeline (e2e-triage).

---

Next steps:

• /fs-code — agent creates a PR to implement this issue
• /fs-code <your instruction> — agent implements with your specific guidance

[fullsend-ai-triage]

🤖 Finished E2e Triage · ✅ Success · Started 10:32 AM UTC · Completed 10:58 AM UTC

Commit: 6697e15 · View workflow run →

[fullsend-ai-triage]

Triage Summary

Workspace	Category	Root Cause	Tests	Issue
app-defaults-app-next	product_bug	inherit-missing-plugin	1	—
argocd	test_fix	route-wait-race	1	#3067
bulk-import	test_fix	module-legacy-merge	2	#3303
bulk-import-orchestrator	product_bug	orchestrator-init-deps	1	#3083
homepage	test_fix	dpdy-version-mismatch	1	#3084
intelligent-assistant	test_fix	lightspeed-rename-gap	2	#3043
orchestrator	product_bug	orchestrator-init-deps	1	#3065
quickstart	test_fix	quickstart-disabled-config	2	#3044
scorecard-filecheck	product_bug	filecheck-github-auth	1	#3304