fix: fast-fail copilot harness when LLM invocation cap is saturated by Copilot · Pull Request #45827 · github/gh-aw

Copilot · 2026-07-15T21:39:02Z

When a run exhausts the per-run LLM invocation cap (CAPIError: 429 Maximum LLM invocations exceeded (N/N)), the pooled budget is shared across all retry attempts in the Execute step. Every --continue retry therefore re-fails instantly with 0B output — the retries are provably useless but the harness kept looping until manually cancelled.

Root cause

harness_retry_guard.cjs already detected maxRunsExceeded via MAX_RUNS_EXCEEDED_PATTERNS, but copilot_harness.cjs never checked that field — only aiCreditsExceeded and awfAPIProxyBlockingRequests were wired into the guard check.

Changes

`copilot_harness.cjs`

Moved nonRetryableGuard computation before classifyCopilotFailure so isInvocationCapExceeded = nonRetryableGuard.maxRunsExceeded is available for classification and the failure log line
Added isInvocationCapExceeded to the guard-check block — breaks out of the retry loop immediately with a clear message indicating which attempt's partial output is preserved
Added isInvocationCapExceeded to the diagnostic log fields

`classifyCopilotFailure`

New isInvocationCapExceeded input → returns "invocation_cap_exceeded" (takes precedence over capi_quota_exceeded)

`detect_agent_errors.cjs`

New INVOCATION_CAP_EXCEEDED_PATTERN matches both the CAPI form (CAPIError: 429 Maximum LLM invocations exceeded (N/N)) and the Anthropic JSON form ("type":"max_runs_exceeded")
Wired into detectErrors() and writeOutputs() as a new invocation_cap_exceeded GitHub Actions output variable

Tests

13 new test cases: pattern matching for both error forms, retry-policy fast-fail, classifyCopilotFailure classification and precedence over capi_quota_exceeded, and detectErrors() output integrity

Before: attempt 1 exhausts 25/25 invocations → attempts 2–4 each hit 429 (25/25) with 0B output → job stalls until manually cancelled
After: attempt 1 exhausts 25/25 → guard fires → harness logs "LLM invocation cap saturated — pooled per-run budget fully exhausted; retries cannot make progress (attempt 1 partial output preserved as run result)" → exits immediately

When a run hits the per-run LLM invocation cap (`CAPIError: 429 Maximum LLM invocations exceeded (N/N)`) the pooled budget is already saturated — subsequent `--continue` retries instantly re-fail with 0B of new output and cannot make progress. Changes: - copilot_harness.cjs: move `nonRetryableGuard` computation before `classifyCopilotFailure` so `isInvocationCapExceeded` is available for classification and the failure log; wire `nonRetryableGuard.maxRunsExceeded` into the guard-check block so the harness breaks out of the retry loop with a clear message instead of continuing to exhausted retries. - classifyCopilotFailure: add `isInvocationCapExceeded` → returns `"invocation_cap_exceeded"` (outranks `capi_quota_exceeded`). - detect_agent_errors.cjs: add `INVOCATION_CAP_EXCEEDED_PATTERN` that matches both the CAPI form and the Anthropic JSON `max_runs_exceeded` form; wire it into `detectErrors()` and `writeOutputs()` as a new `invocation_cap_exceeded` GitHub Actions output variable; update the file-level JSDoc. - Tests: add 13 new test cases covering pattern matching, retry-policy fast-fail for both CAPI and Anthropic forms, `classifyCopilotFailure` classification and precedence, and `detectErrors()` output. Closes #44788 Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

…ssage and clarify test name Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot

Pull request overview

Fast-fails Copilot retries when the shared LLM invocation cap is exhausted.

Changes:

Adds invocation-cap detection and classification.
Stops retries immediately and improves diagnostics.
Adds pattern and retry-policy tests.

Show a summary per file

File	Description
`actions/setup/js/copilot_harness.cjs`	Adds fast-fail guard and classification.
`actions/setup/js/copilot_harness.test.cjs`	Adds retry and classification tests.
`actions/setup/js/detect_agent_errors.cjs`	Detects and emits invocation-cap errors.
`actions/setup/js/detect_agent_errors.test.cjs`	Tests supported error forms and detection results.

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Files reviewed: 4/4 changed files
Comments generated: 3
Review effort level: Medium

    `model_not_supported_error=${results.modelNotSupportedError}`,
    `http_400_response_error=${results.http400ResponseError}`,
    `capi_quota_exceeded_error=${results.capiQuotaExceededError}`,
+    `invocation_cap_exceeded=${results.invocationCapExceeded}`,


          if (nonRetryableGuard.aiCreditsExceeded) reasons.push("AI credits budget exceeded");
          if (nonRetryableGuard.awfAPIProxyBlockingRequests) reasons.push("AWF API proxy is blocking requests");
+          if (isInvocationCapExceeded) {
+            const budgetMsg = result.hasOutput ? `attempt ${attempt + 1} partial output preserved as run result` : "no output produced";


      if (result.exitCode === 0) return false;
      if (hasNumerousPermissionDeniedIssues(result.output)) return false;
      if (isCAPIQuotaExceededError(result.output)) return false;
+      if (detectNonRetryableHarnessGuard(result.output).maxRunsExceeded) return false;


github-actions · 2026-07-16T01:03:05Z

🤖 PR Triage

Field	Value
Category	`bug`
Risk	`low`
Score	65 / 100
Action	`fast_track`
Batch	`js-runtime-fixes` (with #45826, #45825)

Score breakdown: Impact 30 + Urgency 22 + Quality 13

Adds fast-fail when LLM invocation cap is hit (429 CAPI), preventing useless retries that waste runner time. JS-layer change with tests. CI passing (17/17). Part of JS runtime fixes batch.

Run §29462479491

Generated by 🔧 PR Triage Agent · 189.6 AIC · ⌖ 5.52 AIC · ⊞ 5.6K · ◷

github-actions · 2026-07-16T04:09:31Z

✅ Design Decision Gate 🏗️ completed the design decision gate check.

No ADR enforcement needed: PR #45827 does not have the 'implementation' label and has 0 new lines of code in business logic directories (threshold: 100).

github-actions · 2026-07-16T04:09:40Z

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

github-actions · 2026-07-16T04:09:40Z

✅ PR Code Quality Reviewer completed the code quality review.

github-actions · 2026-07-16T04:09:48Z

✅ Test Quality Sentinel completed test quality analysis.

github-actions · 2026-07-16T04:16:25Z

🧪 Test Quality Sentinel Report

⚠️ Test Quality Score: 75/100 — Acceptable

Analyzed 10 test(s): 10 design, 0 implementation, 0 violation(s).

📊 Metrics (10 tests)

Metric	Value
Analyzed	10 (Go: 0, JS: 10)
✅ Design	10 (100%)
⚠️ Implementation	0 (0%)
Edge/error coverage	10 (100%)
Duplicate clusters	1
Inflation	YES — both test files exceed 2:1 ratio
🚨 Violations	0

Test	File	Classification	Issues
INVOCATION_CAP_EXCEEDED_PATTERN — CAPI 429 form	detect_agent_errors.test.cjs	design_test / high_value	—
INVOCATION_CAP_EXCEEDED_PATTERN — embedded in log	detect_agent_errors.test.cjs	design_test / high_value	—
isInvocationCapExceededError — Anthropic JSON form	detect_agent_errors.test.cjs	design_test / high_value	—
isInvocationCapExceededError — human-readable form	detect_agent_errors.test.cjs	design_test / high_value	—
isInvocationCapExceededError — case-insensitive	detect_agent_errors.test.cjs	design_test / high_value	—
isInvocationCapExceededError — does not match unrelated	detect_agent_errors.test.cjs	design_test / high_value	—
does not retry — CAPI form (cap saturated)	copilot_harness.test.cjs	design_test / high_value	—
does not retry — Anthropic JSON form (cap saturated)	copilot_harness.test.cjs	design_test / high_value	minor dup pattern with detect_agent_errors
classifies as invocation_cap_exceeded	copilot_harness.test.cjs	design_test / high_value	—
invocation_cap_exceeded outranks capi_quota_exceeded	copilot_harness.test.cjs	design_test / high_value	—

⚠️ Flagged Tests (2) — inflation only, no action required

copilot_harness.test.cjs — +35 test lines vs +11 prod lines (ratio 3.2:1). Acceptable: the new production code is a compact fast-fail guard; tests must cover multiple error-message variants. No blocking issue.

detect_agent_errors.test.cjs — +76 test lines vs +34 prod lines (ratio 2.2:1). Marginally above the 2:1 threshold. Multiple positive/negative regex variants across 5 sub-tests for INVOCATION_CAP_EXCEEDED_PATTERN justify the volume.

Verdict

✅ Passed. 0% implementation tests (threshold: 30%). Score penalty from test inflation (both files >2:1), but all new tests are high-value behavioral contracts verifying fast-fail LLM invocation-cap logic end-to-end: pattern matching, isInvocationCapExceededError helper, retry suppression, and failure classification priority.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

🧪 Test quality analysis by Test Quality Sentinel · 44.3 AIC · ⌖ 10.4 AIC · ⊞ 6.8K · ◷
_{Comment /review to run again}

pelikhan · 2026-07-16T07:54:56Z

@copilot merge main and recompile
Fix https://github.com/github/gh-aw/actions/runs/29480441627/job/87562713176

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot · 2026-07-16T08:18:20Z

@copilot merge main and recompile
Fix https://github.com/github/gh-aw/actions/runs/29480441627/job/87562713176

Merged main (already up to date), recompiled, and fixed the failing CGO test job in 11701d8.

pelikhan · 2026-07-16T08:26:59Z

@copilot fix https://github.com/github/gh-aw/actions/runs/29482801860/job/87570627774?pr=45827

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot · 2026-07-16T08:46:38Z

@copilot fix https://github.com/github/gh-aw/actions/runs/29482801860/job/87570627774?pr=45827

Fixed in 1084d63. I investigated the failing CGO job logs and updated the stale test expectations to match the current embedded action/container pins (gh-aw-mcpg:v0.4.1, actions/setup-node@v7, and a known existing container in spec tests). Targeted package tests now pass for the failing cases.

gh-aw-bot · 2026-07-16T09:11:02Z

@copilot please run the pr-finisher skill, address unresolved review comments, fix any failing checks, and rerun checks once the branch is up to date.

Generated by 👨‍🍳 PR Sous Chef · 8.77 AIC · ⌖ 6.55 AIC · ⊞ 5.1K · ◷
_{Comment /souschef to run again}

…-on-retry Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

Copilot · 2026-07-16T09:28:03Z

@copilot please run the pr-finisher skill, address unresolved review comments, fix any failing checks, and rerun checks once the branch is up to date.

...

Completed a pr-finisher pass on this branch.

Merged main into this PR in b57d59fcbd (branch is now up to date with base).
Re-checked unresolved review feedback: the remaining points are already covered by existing changes in this branch (no additional code edits were needed in this pass).
Investigated current checks via Actions API: no failed jobs in the latest PR run snapshot (run_id=29484652912, failed_jobs=0).
Re-ran local validation successfully after the merge: make fmt, make lint, make test-unit, make test, make recompile.

I can’t trigger fresh CI from this cloud-agent path; a maintainer re-trigger is still required (close/reopen PR, workflow_dispatch, or maintainer push).

pelikhan · 2026-07-16T09:54:03Z

@copilot merge main recompile fix conflicts

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot · 2026-07-16T10:29:09Z

@copilot merge main recompile fix conflicts

Merged main, resolved conflicts, and recompiled in 0524f1c.

gh-aw-bot · 2026-07-16T10:43:17Z

@copilot please run the pr-finisher skill on this PR. Address the unresolved review feedback and then rerun checks.

Unresolved review threads (newest first):

github-actions: fix: fast-fail copilot harness when LLM invocation cap is saturated #45827 (comment)
github-actions: fix: fast-fail copilot harness when LLM invocation cap is saturated #45827 (comment)
github-actions: fix: fast-fail copilot harness when LLM invocation cap is saturated #45827 (comment)
github-actions: fix: fast-fail copilot harness when LLM invocation cap is saturated #45827 (comment)
copilot-pull-request-reviewer: fix: fast-fail copilot harness when LLM invocation cap is saturated #45827 (comment)
copilot-pull-request-reviewer: fix: fast-fail copilot harness when LLM invocation cap is saturated #45827 (comment)
copilot-pull-request-reviewer: fix: fast-fail copilot harness when LLM invocation cap is saturated #45827 (comment)

Please reply in-thread where appropriate and resolve the addressed threads once the fixes land.

Generated by 👨‍🍳 PR Sous Chef · 9.88 AIC · ⌖ 6.21 AIC · ⊞ 7.5K · ◷
_{Comment /souschef to run again}

Initial plan

9b36021

Copilot AI assigned Copilot and pelikhan Jul 15, 2026

Copilot started work on behalf of pelikhan July 15, 2026 21:39 View session

Copilot AI linked an issue Jul 15, 2026 that may be closed by this pull request

Copilot harness retries max-turns/invocation-cap exhaustion against a saturated pooled budget (guaranteed instant re-fail; should fast-fail) #44788

Closed

Copilot AI and others added 2 commits July 15, 2026 22:00

fix: address review feedback - use actual attempt number in budget me…

3733aa5

…ssage and clarify test name Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Fix fast-fail for saturated invocation budget in Copilot harness~~ fix: fast-fail copilot harness when LLM invocation cap is saturated Jul 15, 2026

Copilot finished work on behalf of pelikhan July 15, 2026 22:03

Copilot AI requested a review from pelikhan July 15, 2026 22:03

pelikhan marked this pull request as ready for review July 15, 2026 22:07

Copilot AI review requested due to automatic review settings July 15, 2026 22:07

Copilot started reviewing on behalf of pelikhan July 15, 2026 22:07 View session

Copilot AI reviewed Jul 15, 2026

View reviewed changes

github-actions Bot mentioned this pull request Jul 16, 2026

[pr-sous-chef] Run report — 0 nudged, 9 skipped #45855

Open

github-actions Bot added pr-action:fast_track pr-agent:copilot pr-priority:medium pr-risk:low pr-type:bug pr-batch:js-runtime-fixes labels Jul 16, 2026

This was referenced Jul 16, 2026

fix: add maxLength: 65536 to body/content fields to exempt from SM-IS-01 10 KB cap #45826

Merged

fix(update_project): parse string-typed fields instead of iterating over characters #45825

Merged

github-actions Bot mentioned this pull request Jul 16, 2026

[aw] No-Op Runs #43499

Open

Merge branch 'main' into copilot/fix-fast-fail-on-retry

bc358c9

Copilot started work on behalf of pelikhan July 16, 2026 07:55 View session

Copilot AI and others added 3 commits July 16, 2026 08:01

chore: plan merge main and recompile fix

7050b1d

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Merge branch 'main' into copilot/fix-fast-fail-on-retry

f744e7a

test: align firewall/action-pin tests with latest embedded pins

11701d8

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot finished work on behalf of pelikhan July 16, 2026 08:18

Copilot started work on behalf of pelikhan July 16, 2026 08:27 View session

Copilot AI and others added 2 commits July 16, 2026 08:31

chore: start investigation for CI failure fix

41efce3

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

test: update action pin expectations for latest embedded versions

1084d63

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot finished work on behalf of pelikhan July 16, 2026 08:47

Copilot started work on behalf of gh-aw-bot July 16, 2026 09:11 View session

Merge remote-tracking branch 'origin/main' into copilot/fix-fast-fail…

b57d59f

…-on-retry Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

Copilot finished work on behalf of gh-aw-bot July 16, 2026 09:28

Copilot started work on behalf of pelikhan July 16, 2026 09:54 View session

Copilot AI and others added 2 commits July 16, 2026 10:12

Merge main and resolve test conflicts

e0e68b0

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

chore: merge main, resolve conflicts, and recompile workflows

0524f1c

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot finished work on behalf of pelikhan July 16, 2026 10:29

Copilot started work on behalf of gh-aw-bot July 16, 2026 10:43 View session

pelikhan merged commit b782292 into main Jul 16, 2026
37 of 38 checks passed

pelikhan deleted the copilot/fix-fast-fail-on-retry branch July 16, 2026 10:45

Copilot stopped work on behalf of gh-aw-bot due to an error July 16, 2026 10:45
The session was cancelled by the user.

Uh oh!

Conversation

Copilot AI commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Root cause

Changes

copilot_harness.cjs

classifyCopilotFailure

detect_agent_errors.cjs

Tests

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Review details

Uh oh!

github-actions Bot commented Jul 16, 2026

🤖 PR Triage

Uh oh!

github-actions Bot commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 16, 2026

🧪 Test Quality Sentinel Report

Verdict

Uh oh!

pelikhan commented Jul 16, 2026

Uh oh!

Copilot AI commented Jul 16, 2026

Uh oh!

pelikhan commented Jul 16, 2026

Uh oh!

Copilot AI commented Jul 16, 2026

Uh oh!

gh-aw-bot commented Jul 16, 2026

Uh oh!

Copilot AI commented Jul 16, 2026

Uh oh!

pelikhan commented Jul 16, 2026

Uh oh!

Copilot AI commented Jul 16, 2026

Uh oh!

gh-aw-bot commented Jul 16, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Copilot AI commented Jul 15, 2026 •

edited

Loading

`copilot_harness.cjs`

`classifyCopilotFailure`

`detect_agent_errors.cjs`

github-actions Bot commented Jul 16, 2026 •

edited

Loading

github-actions Bot commented Jul 16, 2026 •

edited

Loading

github-actions Bot commented Jul 16, 2026 •

edited

Loading

github-actions Bot commented Jul 16, 2026 •

edited

Loading