Safe Output Health Report — 2026-03-28

[github-actions[bot]]

Executive Summary

• Period: Last 24 hours (2026-03-28)
• Runs Analyzed: 17 (recent 24h window)
• Workflows Active: 15 distinct workflows
• Safe Output Messages Processed: 24
• Safe Output Messages Succeeded: 23
• Safe Output Messages Failed: 1 (+ 1 cancelled due to cascade)
• Safe Output Warnings: 1 (label missing)
• Safe Output Success Rate: 95.8%
• Detection Failures Blocking Safe Outputs: 4 runs
• Error Clusters Identified: 4 clusters

Overall system health is good. The single hard safe output failure (EP019 variant in Smoke Claude) is a known pattern related to protected path enforcement. The 4 detection failures blocking safe outputs are a more significant concern—they silently discard valid agent work.

---

Safe Output Job Statistics

Job Type	Executions	Failures	Warnings	Success Rate
add_comment	7	0	0	100%
create_discussion	2	0	1	100% (warn)
noop	2	0	0	100%
push_to_pull_request_branch	1	1	0	0%
create_issue	1	0¹	0	100%
dispatch_workflow	1	0	0	100%
set_issue_type	1	0	0	100%
missing_tool	1	0	0	100% (informational)
add_reviewer	1	0	0	100%
update_pull_request	1	0	0	100%
create_pull_request_review_comment	1	0	0	100%
submit_pull_request_review	1	0	0	100%

¹ create_issue was cancelled due to the preceding push_to_pull_request_branch failure.

Runs that executed safe outputs successfully: Issue Monster (×2), Daily Firewall, PR Triage Agent, Constraint Solving, Weekly Editors Health Check, Smoke Copilot, Smoke Claude (partial)

Runs where safe outputs were SKIPPED due to detection failure: Smoke Create Cross-Repo PR, Smoke Update Cross-Repo PR, Auto-Triage Issues, Smoke Gemini

---

Error Clusters

Cluster 1: Copilot Detection Authentication Failures (EP024 — HIGH)

• Count: 3 authentication failures + 1 truncated output = 4 detection failures today
• Affected Jobs: detection (blocks safe_outputs)
• Affected Workflows: Smoke Create Cross-Repo PR, Smoke Update Cross-Repo PR, Auto-Triage Issues, Smoke Gemini
• Sample Errors:

  Error: Authentication failed (Request ID: 4004:965CC:C43FAB:FE7892:69C7D214)
  ##[error]Process completed with exit code 1.
  ##[error]❌ Failed to parse detection result: No THREAT_DETECTION_RESULT found in detection log.
  ```
  For Smoke Gemini (truncated output):
  ```
  "content": "THREAT_DETECTION_RESULT:{\"prompt_injection\":false,\"secret_leak\":false,\"malicious_",
  "delta": true
  ```
  

• Root Cause: The Copilot detection model either fails to authenticate (3 cases with different request IDs) or produces a streaming response that gets truncated before completing the required output format. The delta:true marker indicates the final message was a streaming delta that was cut off mid-JSON.
• Impact: HIGH — 4 workflows had their safe outputs silently blocked. Valid agent outputs (PR patches, issue labels, discussion posts) were discarded.
• Pattern History: First observed 2026-03-26 (4 cases), recurring today 2026-03-28 (4 cases). Now 8 total occurrences over 3 days.

Cluster 2: push_to_pull_request_branch Protected Path Violation (EP019b — NEW)

• Count: 1 occurrence
• Affected Jobs: safe_outputs (push_to_pull_request_branch handler)
• Affected Workflow: Smoke Claude (run-23685128220)
• Sample Error:

  ##[error]Cannot push to pull request branch: patch modifies protected files (.github/smoke-test-push-23685128220).
  Add them to the allowed-files configuration field or set protected-files: fallback-to-issue.
  ##[error]✗ Message 8 (push_to_pull_request_branch) failed
  ##[warning]⚠️ Code push operation 'push_to_pull_request_branch' failed — remaining safe outputs will be cancelled
  ⏭ Message 9 (create_issue) cancelled
  

• Root Cause: Smoke Claude's agent wrote a code patch targeting .github/smoke-test-push-{run_id}, which falls under the protected path prefix .github/. The handler correctly blocks this, but causes cascade cancellation of the following create_issue message.
• Impact: MEDIUM — Safe outputs job reports as failed. Issue creation (smoke test result) was not saved.

Cluster 3: EP025 actions/setup Not Found — Infrastructure (Multiple Jobs)

• Count: 5 occurrences across 5 different runs
• Affected Jobs: push_repo_memory, upload_assets, agent (post-cleanup)
• Affected Workflows: PR Triage Agent, Daily Firewall Logs, Smoke Create Cross-Repo PR, Smoke Update Cross-Repo PR, Auto-Triage Issues
• Sample Error:

  ##[error]Can't find 'action.yml', 'action.yaml' or 'Dockerfile' under '/home/runner/work/gh-aw/gh-aw/actions/setup'.
  Did you forget to run actions/checkout before running your local action?
  

• Root Cause: This is a checkout ordering issue — the actions/setup local action is referenced before the repository is fully checked out in certain job configurations. Affects push_repo_memory, upload_assets, and agent post-cleanup steps.
• Impact: MEDIUM — These are NOT safe output job failures, but they cause workflow conclusion to show failure even when the safe_outputs job itself ran correctly (e.g., PR Triage Agent: safe output noop succeeded, but workflow failed due to push_repo_memory). Also contributes to the detection failures seen in Cluster 1.
• Pattern History: First observed 2026-03-26 (2 cases). Now 5 cases today. Escalating.

Cluster 4: EP_LABELS_MISSING — Constraint Solving Labels (4th Consecutive Day)

• Count: 1 occurrence (4th consecutive day: 2026-03-25 to 2026-03-28)
• Affected Jobs: safe_outputs (create_discussion labels step)
• Affected Workflow: Constraint Solving — Problem of the Day
• Sample Warning:

  ##[warning]Could not find label IDs for: constraint-solving, problem-of-the-day
  These labels may not exist in the repository. Available labels: bug, documentation, ...
  ##[warning]⚠ No matching labels found in repository for: constraint-solving, problem-of-the-day
  

• Root Cause: Labels constraint-solving and problem-of-the-day are referenced in the workflow frontmatter but don't exist in the repository. The create_discussion itself succeeds, but labels are not applied.
• Impact: LOW — Non-fatal warning. Discussion is created successfully; only labels are missing.

---

Root Cause Analysis

Detection Job Issues (Blocking Safe Outputs)

The EP024 escalation is the most systemic concern today. Three Copilot detection runs failed with authentication errors — different request IDs in each case, suggesting intermittent token/credential issues rather than a systematic misconfiguration. One run (Smoke Gemini) shows a streaming truncation pattern where the detection model starts producing output but the response ends before completing the JSON. This is likely a streaming timeout or network interruption.

The detection failures propagate as a blocker: safe_outputs.if evaluates needs.detection.result == 'success', so any detection failure silently skips all safe outputs, discarding valid agent work.

Safe Output Handler Issues

The single safe output failure (EP019b) is a correct enforcement of security policy — the agent wrote a file to .github/smoke-test-push-{run_id}, which is a protected path. However, the cascade behavior (cancelling subsequent create_issue) is overly aggressive for a smoke test workflow where safe output writes are expected to be low-risk.

Infrastructure Issues (EP025)

The actions/setup not-found errors appear in diverse job types, suggesting that after certain checkout operations (like checking out a memory branch or performing sparse checkout), the main repository's actions/ directory is no longer available. This is a post-checkout state contamination bug.

---

Recommendations

Critical Issues (Immediate Action Required)

1. EP024: Copilot Detection Authentication Failures
   • Priority: High
   • Root Cause: Intermittent Copilot token auth failures during detection job; streaming truncation in at least 1 case
   • Recommended Action:
     • Add retry logic (1-2 retries with backoff) in the detection step when authentication fails
     • For streaming truncation: add a post-processing check that validates the THREAT_DETECTION_RESULT line is complete before considering the detection done
     • Monitor: If auth failures persist, investigate whether the Copilot token needs rotation
   • Affected: Smoke Create Cross-Repo PR, Smoke Update Cross-Repo PR, Auto-Triage Issues, Smoke Gemini

Bug Fixes Required

2. EP019b: Smoke Claude Agent Writing to .github/ Path

   • Priority: Medium
   • Location: Smoke Claude workflow prompt / push_to_pull_request_branch config
   • Problem: Agent patches target .github/smoke-test-push-{run_id}, a protected prefix
   • Fix:
     • Option A: Configure protected-files: fallback-to-issue in smoke-claude's push_to_pull_request_branch handler config so a review issue is created instead of hard failure
     • Option B: Instruct the smoke test agent to write artifacts to non-protected paths (/tmp/ or smoke-tests/)
   • Affected Jobs: push_to_pull_request_branch, create_issue (cascade)

3. EP025: actions/setup Not Found After Branch Checkout

   • Priority: Medium
   • Location: push_repo_memory, upload_assets job definitions
   • Problem: Post-checkout state loses the actions folder, breaking local action references
   • Fix: Add a Checkout actions folder step before any local action invocation in push_repo_memory and upload_assets jobs (similar to how the safe_outputs job does it)
   • Affected: 5+ workflows, escalating from 2 to 5 occurrences in 2 days

Configuration Changes

4. EP_LABELS_MISSING: Create Missing Labels in Repository
   • Priority: Low
   • Current: Labels constraint-solving and problem-of-the-day do not exist in github/gh-aw
   • Recommended: Create these two labels in the repository, or remove them from Constraint Solving workflow frontmatter
   • Reason: 4th consecutive day of this warning; simple fix

---

Work Item Plans

Work Item 1: Detection Retry Logic for Authentication Failures

• Type: Bug Fix
• Priority: High
• Description: The Copilot detection job fails with intermittent authentication errors, causing valid agent safe outputs to be silently discarded. A retry mechanism would make detection more robust.
• Acceptance Criteria:
  • Detection job retries on authentication failure (max 2 retries, 10s backoff)
  • Streaming truncation detected and handled (re-run detection if output is incomplete)
  • Detection failures log actionable error messages with retry count
• Technical Approach: Modify the detection entrypoint to wrap the model call with retry logic; add output validation before setting detection_success=true
• Estimated Effort: Medium
• Dependencies: Access to detection job entrypoint code

Work Item 2: Fix EP025 actions/setup Availability in push_repo_memory and upload_assets

• Type: Bug Fix
• Priority: Medium
• Description: Various jobs fail because actions/setup is not available after branch checkout operations corrupt the working directory state.
• Acceptance Criteria:
  • push_repo_memory job reliably finds actions/setup
  • upload_assets job reliably finds actions/setup
  • No more Can't find 'action.yml' errors in these jobs
• Technical Approach: Add a Checkout actions folder step (sparse checkout of the actions/ directory) before any local action reference in affected jobs
• Estimated Effort: Small
• Dependencies: Identify all jobs affected by this pattern

Work Item 3: Fix Smoke Claude Agent Path for push_to_pull_request_branch (EP019b)

• Type: Configuration Fix
• Priority: Medium
• Description: Smoke Claude creates patches targeting .github/ paths, triggering the protected-file block and cascading failure.
• Acceptance Criteria:
  • Smoke Claude smoke test writes do not target .github/ protected prefix
  • If a protected-path push is attempted, fallback creates a review issue instead of hard failure
  • Subsequent safe output messages are not cancelled
• Technical Approach: Add protected-files: fallback-to-issue to Smoke Claude's push_to_pull_request_branch config; update agent instructions to use non-protected paths for smoke test artifacts
• Estimated Effort: Small

---

Historical Context

Recent Trend (Last 7 Days)

Date	Success Rate	Hard Failures	Warnings	Main Pattern
2026-03-22	N/A	2	0	EP006 + EP022 candidate
2026-03-23	100%	0	0	Clean run
2026-03-24	97.0%	1	0	EP022: push target branch missing
2026-03-25	100%	0	1	EP_LABELS_MISSING (2nd day)
2026-03-26	N/A	0	0	EP024 first observed (detection), EP025 first observed
2026-03-27	100%	0	1	EP_LABELS_MISSING (3rd day), clean safe outputs
2026-03-28	95.8%	1	1	EP024 escalating (4 detection failures), EP025 escalating (5 occurrences), EP019b new

Trend: Success rate stable at 95-100% for safe output operations. However, detection failures (EP024) are an escalating pattern that silently discards agent work — first 4 cases on 2026-03-26, then 4 more today.

EP025 escalation: Went from 2 occurrences (2026-03-26) to 5 occurrences today, now affecting 5+ workflows.

---

Metrics and KPIs

• Overall Safe Output Success Rate: 95.8% (23/24 messages)
• Most Reliable Job Types: add_comment, create_discussion, noop, dispatch_workflow (100%)
• Most Problematic Job Type: push_to_pull_request_branch (0% today — 1/1 blocked by EP019b)
• Detection Failure Rate: 4/17 runs (23.5%) had detection fail, blocking safe outputs
• Infrastructure Failure Rate: 5/17 runs (29.4%) had EP025 actions/setup failures in non-safe-output jobs

---

Next Steps

• Investigate and fix EP024: Add retry logic to Copilot detection authentication
• Fix EP025: Add Checkout actions folder step to push_repo_memory and upload_assets jobs
• Fix EP019b: Configure protected-files: fallback-to-issue in smoke-claude workflow
• Create missing labels constraint-solving and problem-of-the-day in repository

References:

• §23685128220 — Smoke Claude (push_to_pull_request_branch failure)
• §23685784887 — Smoke Create Cross-Repo PR (EP024 auth failure)
• §23685447411 — Smoke Gemini (EP024 truncated output)

AI generated by Safe Output Health Monitor · history

• expires on Mar 29, 2026, 1:20 PM UTC

[github-actions[bot]]

💥 KAPOW! The Smoke Test Agent was HERE! 🦸

WHOOSH! — Claude swooped in, ran all the checks, and — WHAM! — every system came back nominal!

🔥 KABOOM! Run §23698148563 complete — the Claude engine is ALIVE and kicking!

ZAP! See you next time, heroes! 💥

💥 [THE END] — Illustrated by Smoke Claude · ◷

[github-actions[bot]]

🤖 Beep boop! The smoke test agent was here! 🚀

Just popped in to say hello from run §23698239359. All systems nominal (mostly)!

"In a world full of bugs, be a smoke test." — Anonymous CI pipeline

👋 Carry on, nothing to see here... except a passing test suite! ✅

📰 BREAKING: Report filed by Smoke Copilot · ◷