Skip to content

[architect] v2-tests aggregate 'test' job fails closed in concurrency-cancelled twin runs, permanently poisoning the head SHA with a red required check #6293

Description

@kubestellar-hive

Architecture Finding

Type: anti-pattern (CI reliability / fail-closed aggregate vs cancel-in-progress)
Affected area: .github/workflows/v2-tests.yml (aggregate test job, concurrency group)

Observed

Duplicate pull_request events for the same head SHA spawn two v2-tests runs in the same concurrency group (v2 Tests-<pr>, cancel-in-progress: true). One twin cancels the other. In the cancelled twin, every shard ends cancelled, but the aggregate test job runs anyway (if: always()) and fails closed — landing a failed check run named test on the same head SHA that the surviving twin later marks green.

Evidence (PR #6257, SHA ef60120c):

  • run 34263090424 (pull_request, cancelled): shards cancelled, aggregate test job conclusion = failure
  • run 34263091584 (pull_request, same second, same SHA): the surviving twin

Same pattern on SHA 8a61d52d (runs 34252781612 fail / 34252782506 success) and on PRs #6185, #6156, #6142, #6132. Tide then reports "Job test has not succeeded" and every one of these PRs is stuck red with no agent-accessible retrigger (gh run rerun not allowlisted, POST rerun blocked by ACMM proxy) — the only recourse is pushing a new commit, which rolls the same dice.

Impact

Any PR can be permanently blocked by a race it did not cause. Hold-gated agent PRs accumulate as "failing CI" and get re-kicked every cycle. Runner minutes are wasted on retrigger pushes.

Recommendation

Change the aggregate job's guard from if: always() to if: '!cancelled()' so a concurrency-cancelled run reports test as skipped/cancelled rather than failed — the surviving run on the same SHA still provides the fail-closed green/red verdict, so no coverage is lost. Optionally also investigate why two pull_request runs are created in the same second for one push (event double-delivery).

Note: an agent cannot PR this — the App token lacks workflows permission to push .github/workflows changes — so this needs a human commit.


Filed by architect agent (ACMM L5 — hold-gated mode)

🐝 Hive Agent: architect | Instance: hosted-available-oke-11-placeholder-r05x | SHA: 475c05ef

— hive: agent=architect backend=copilot model=claude-opus-4-6

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent/architectCreated or modified by the architect agentarchitectureApproved by a Hive merger/owner for auto-merge on green CIhelp wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.hive/hosted-available-oke-11-placeholder-r05xApproved by a Hive merger/owner for auto-merge on green CItech-debtApproved by a Hive merger/owner for auto-merge on green CI

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions