[aw-failures] [aw] Failure Investigator Report — 2026-07-30 (6h)

[github-actions]

Executive Summary

Fix the shell-expansion permission gate — it's blocking the officially-documented jq -Rs "$FILE" pattern and burning agents' entire invocation budget on denied retries. Design Decision Gate hit Maximum LLM invocations exceeded (20/20) after 5 different retries of a blocked file-write pattern plus 10 stacked 429 rate-limit backoffs — no PR comment was ever posted. Fix the evals push race next — push_evals_state lost a 4-attempt rebase battle over evals.jsonl and dropped this run's results entirely.

Two new failure gaps found this cycle, one P0 and one P1. Both are new — neither has existing tracking coverage, so this report is a new parent (previous day's tracking issues don't cover either signature).

Failure Cluster Table

Cluster	Severity	Workflow	Run	Signature	Status
A	P0	Design Decision Gate 🏗️	§30520617010	Permission-denied retry loop on "$FILE" path expansion → invocation cap exhausted (20/20)	New — sub-issue filed
B	P1	Test Quality Sentinel	§30520616983	push_evals_state rebase conflict in evals.jsonl, 4/4 push attempts failed	New — documented here, not yet ticketed (create-issue budget capped at 2 this cycle)
C	P2	PR Code Quality Reviewer	§30518853718	Execute GitHub Copilot CLI step failure	Matched to existing #49022 — no action
D	P2	Daily Container Image Security Scan	§30518927510	Gate correctly failed on Critical vulnerabilities detected in container images	Expected behavior, not a workflow bug — no action
E	P2	CLI Version Checker	§30518846175	Generic exit code 1, 130k tokens / 24 turns, no timeout, no rate-limit or permission signal in logs or audit	Isolated, insufficient signal — no action, watch for recurrence

Evidence

Cluster A — Design Decision Gate invocation-cap exhaustion

Run §30520617010, PR #49066. The agent tried to post an ADR-required comment via safeoutputs add_comment and got denied 5 different ways, all variants of the same officially-recommended jq -Rs file-injection pattern:

FILE=/tmp/gh-aw/agent/comment-body.md && jq -Rs --argjson pr 49066 '{pull_request_number: $pr, body: .}' "$FILE"
→ Error: Shell expansion syntax in paths requires manual approval

After exhausting retry variants (heredoc, direct file write, piping to safeoutputs, $FILE substitution), the harness hit 10 consecutive 429 rate-limit backoffs (582ms → 39880ms), then terminated: "API Error: Request rejected (429) · Maximum LLM invocations exceeded (20 / 20)". Duration 5m20s, 22 turns, zero comment posted.

audit-diff against the nearest successful run of the same workflow (§30520438606) shows zero firewall/MCP anomalies and zero domain drift — this rules out a sandbox network regression. The failure is purely in the harness's path-expansion permission check colliding with its own recommended safe-outputs pattern, compounded by API rate limiting.

Cluster B — Test Quality Sentinel evals push race

Run §30520616983. agent job succeeded; push_evals_state failed:

CONFLICT (content): Merge conflict in evals.jsonl
##[error]Failed to push evals results after 4 attempts: pushSignedCommits: failed to rebase commit range onto current GraphQL parent

Signed-commit rebase kept losing the race against a concurrent writer to refs/heads/evals/testqualitysentinel — remote HEAD moved between the ls-remote check and the rebase attempt, every time, across all 4 retries.

Existing Issue Correlation

Cluster	Matched issue	Confidence
C (PR Code Quality Reviewer)	#49022 "Harden PR Code Quality Reviewer against background task-tool sub-agent failures"	Medium — same workflow/step, truncated logs too sparse to confirm identical signature, but pattern and timing align
A, B	none	— new gaps
—	#49023 (setup_globals.cjs MODULE_NOT_FOUND)	Not seen in this window — leave open, no evidence either way
—	#49024 (Failure Investigator timeout)	Unrelated to this window's failures — leave open

Fix Roadmap

P0 — Design Decision Gate invocation-cap exhaustion. Stop the harness from denying its own documented jq -Rs "$FILE" pattern, and cap denied-command retries before they burn the invocation budget. Sub-issue filed below.

P1 — Test Quality Sentinel evals push race. Serialize push_evals_state writes (queue/lock) or move to per-run file paths merged in a separate step, since blind rebase-retry doesn't converge under concurrent writers. Not filed as a separate GitHub issue this cycle (create-issue budget capped at 2) — file next cycle if it recurs.

P2 — no action needed. Container scan gate is working as intended; PR Code Quality Reviewer already tracked in #49022; CLI Version Checker is a single isolated occurrence with no diagnostic signal.

Sub-Issues Created

• P0 fix filed as a linked sub-issue (see below) for the Design Decision Gate invocation-cap exhaustion.
• P1 (evals push race) documented above only — deferred to next cycle under the 2-issue create budget.

Generated by 🔍 [aw] Failure Investigator (6h) · agent · 91.7 AIC · ⌖ 22.4 AIC · ⊞ 6.8K · ◷

• expires on Aug 5, 2026, 11:49 PM UTC-08:00

---

Update — 2026-07-30 13:19 UTC (6h window, run §30546089667)

Fix the "Pre-fetch ADR gate PR context" step in Design Decision Gate first — it fails before the agent even starts, so the gate produces zero review output on affected PRs. New sub-issue below.

Fix Daily Max Ai Credits Test's failure handling next — it burns real tokens/AI credits (4,065 tokens, 10.9 AIC) and still ends in conclusion: failure with 0 turns recorded, which means the test's own boundary condition isn't handled gracefully. New sub-issue below.

No new parent needed — no P0 failure in this window lacks existing tracking coverage, so findings are filed as sub-issues under this report per the standing correlation rule.

Failure cluster table

Cluster	Severity	Workflows	Run IDs	Status
Copilot CLI zero-token silent failure	P0	PR Code Quality Reviewer	§30531715045	Already tracked — #49022 (confirmed recurring, evidence added)
Job fails despite real token usage	P1	Daily Max Ai Credits Test	§30539804771	New sub-issue
Pre-fetch step fails before agent starts	P1	Design Decision Gate 🏗️	§30538838918	New sub-issue (distinct from #49096)
Process Safe Outputs step fails, no error text captured	P1	PR Sous Chef	§30531795286	Under-evidenced — held back, see below
Claude CLI loops post-completion until 20-min step timeout	P1	Typist - Go Type Analysis	§30540528905	Under-evidenced — held back, see below
Unresolved workflow identity (3 runs, API timeouts)	Unknown	Unknown — firewall pattern suggests a Go-toolchain/module-download step	30527752090, 30525327827, 30523835792	Investigate separately — see below

Evidence detail

PR Code Quality Reviewer / Copilot CLI (#49022, confirmed recurring)
audit-diff on run 30531715045 shows the Copilot CLI made 32 allowed requests to api.githubcopilot.com but the run recorded zero input/output/cache tokens anywhere — a genuine silent-failure signature (requests go out, nothing comes back usable), not a rate limit or auth block. This matches the exact failure mode #49022 was opened for. No fix landed yet; recommend prioritizing since this is the highest-volume P0 cluster.

Daily Max Ai Credits Test
audit on run 30539804771 shows real usage (input_tokens: 3902, output_tokens: 163, cache_write_tokens: 39832, ai_credits: 10.901) and a clean firewall trace (6/6 allowed to api.githubcopilot.com, 0 blocked) — the Copilot CLI clearly ran and did work. Yet the job step Execute GitHub Copilot CLI still concluded failure with Turns: 0 recorded. Given the workflow's purpose is literally to probe the AI-credits ceiling, hitting that ceiling should be a handled, successful outcome (exit 0, report the number), not a crashed job. Root cause is probably in how the workflow script interprets a credit-limit response from the Copilot CLI. This is mechanically distinct from the PR Code Quality Reviewer case above (real usage recorded vs. zero usage recorded) — do not merge into #49022.

Design Decision Gate 🏗️
audit on run 30538838918 confirms the failure is in Pre-fetch ADR gate PR context, a pre-checkout, pre-agent setup step — every downstream step (checkout, Claude Code CLI, safe outputs) is skipped, and Turns: 0 / TokenUsage: 0 confirms the agent never ran at all. This is mechanically distinct from #49096 (which is about the LLM invocation cap being exhausted during agent execution via permission-denied retries) — that failure mode requires the agent to have started. Run context: triggered by an issue_comment command on PR #49115 (branch copilot/add-support-for-local-paths) — worth checking whether that PR's path-handling changes broke the gate's own pre-fetch script.

PR Sous Chef / Typist — held back
Both failures' captured 50-line tails contain only git-credential cleanup and "Set output" declarations — no exception, status code, or stack trace is visible in the truncated window. Typist's tail does show the agent completing its analysis, writing it to /tmp/gh-aw/agent/typist-discussion-v3.md, then looping on safeoutputs --help calls until GitHub killed the step at the 20-minute mark — a real, if narrow, finding — but a single occurrence with only one signal isn't enough to file a confident, actionable sub-issue without a second audit call, and the audit budget for this run was spent on the two higher-severity, better-evidenced clusters above. Recommend the next investigation pass pull full untruncated logs for these two before filing.

3 unresolved run IDs
30527752090, 30525327827, 30523835792 were in failed_run_ids but a per-run audit call timed out (>60s) before workflow identity could be established. A comparative audit across the three showed a shared firewall footprint (proxy.golang.org, sum.golang.org, storage.googleapis.com, release-assets.githubusercontent.com) consistent with a Go-toolchain/module-download step, and one anomaly: release-assets.githubusercontent.com was denied in run 30527752090 but absent (not attempted) in the other two — a possible intermittent GitHub Actions tool-download flake. Not enough to name the workflow confidently; flag for the next pass rather than guess.

Fix roadmap

• P0 — [aw-failures] Harden PR Code Quality Reviewer against background task-tool sub-agent failures #49022 (PR Code Quality Reviewer Copilot CLI silent failure): no new issue, evidence appended confirming recurrence today.
• P1 — new sub-issue: Daily Max Ai Credits Test crashes instead of reporting the credit ceiling.
• P1 — new sub-issue: Design Decision Gate's pre-fetch step fails before the agent starts (separate from [aw-failures] Design Decision Gate exhausts LLM invocation cap after permission-denied retries on shell-expanded file paths #49096).
• P2 / deferred — PR Sous Chef Process Safe Outputs failure and Typist's post-completion loop/timeout: both need one more full-log pull before a sub-issue can be written with confidence; the 3 unresolved run IDs need workflow-identity resolution.

Sub-issues created this run

• New: Daily Ai Credits Test failure handling (see linked sub-issue)
• New: Design Decision Gate pre-fetch step failure (see linked sub-issue)

References:

• §30531715045 — PR Code Quality Reviewer (confirms [aw-failures] Harden PR Code Quality Reviewer against background task-tool sub-agent failures #49022)
• §30539804771 — Daily Max Ai Credits Test
• §30538838918 — Design Decision Gate 🏗️

Generated by 🔍 [aw] Failure Investigator (6h) · agent · 180.1 AIC · ⌖ 21.7 AIC · ⊞ 6.8K · ◷