Skip to content

Make threat-detect binary install step continue-on-error in warn mode - #52400

Merged
pelikhan merged 6 commits into
mainfrom
copilot/aw-failures-harden-binary-installers
Aug 15, 2026
Merged

Make threat-detect binary install step continue-on-error in warn mode#52400
pelikhan merged 6 commits into
mainfrom
copilot/aw-failures-harden-binary-installers

Conversation

Copilot AI commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

A GitHub Releases CDN blip caused the evals/detection job installers for the AWF firewall and threat-detect binaries to fail across multiple PR-gate workflows within a 25-minute window. The threat-detect binary download already had retry-with-backoff (curl --retry 5 --retry-delay 10 --retry-max-time 180), but a hard failure after retries still marked the whole detection job as failure — even though the job's own logic (GH_AW_DETECTION_CONTINUE_ON_ERROR) already treats a missing threat-detect binary as non-fatal in warn mode.

Compiler change

  • buildInstallThreatDetectStep now takes *WorkflowData and emits continue-on-error: true (or the configured expression) for the "Install threat-detect binary" step, reusing the same continue-on-error resolution already applied to the detection conclusion/analysis steps:
- name: Install threat-detect binary
  if: always() && steps.detection_guard.outputs.run_detection == 'true'
  continue-on-error: true
  run: |
    bash "${RUNNER_TEMP}/gh-aw/actions/install_threat_detect_binary.sh" latest

This keeps the install step's failure semantics consistent with the job's existing tolerance policy: in warn mode a transient download failure no longer flips the job conclusion to failure; in strict mode (continue-on-error: false) the step still fails the job as before.

Test coverage

  • threat_detection_isolation_test.go: added an assertion in TestExternalDetectorPath verifying the install step carries continue-on-error: true in the default warn mode.

Generated workflows

  • Recompiled all .lock.yml files; 104 workflows using features: gh-aw-detection: true pick up the new continue-on-error: true line on this step.

Generated by 👨‍🍳 PR Sous Chef · gpt54 · 26 AIC · ⌖ 5.66 AIC · ⊞ 8.5K ·
Comment /souschef to run again


Run: https://github.com/github/gh-aw/actions/runs/31871584565> Generated by 👨‍🍳 PR Sous Chef · gpt54 · 11.5 AIC · ⌖ 5.77 AIC · ⊞ 8.7K ·

Comment /souschef to run again

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title [WIP] Add retry-with-backoff to AWF/threat-detect binary installers Make threat-detect binary install step continue-on-error in warn mode Aug 13, 2026
Copilot AI requested a review from pelikhan August 13, 2026 03:20
@github-actions

Copy link
Copy Markdown
Contributor

PR Triage

Category: chore | Risk: low | Priority: low | Score: 31/100

Score breakdown: impact 15/50, urgency 8/30, quality 8/20

CI: unknown (draft)

Recommended action: defer

CI hardening chore, still draft/blocked.

Automated triage — run 52400

Generated by 🔧 PR Triage Agent · auto · 67.2 AIC · ⌖ 2.77 AIC · ⊞ 7.8K ·

@github-actions

Copy link
Copy Markdown
Contributor

PR Triage

Category: chore · Risk: low · Priority: low
Score: 30/100 (impact 10, urgency 8, quality 12)
Recommended action: defer

Makes threat-detect binary install step continue-on-error in warn mode. Mostly bulk lock.yml regeneration from a one-line source change; low value/urgency.

Generated by 🔧 PR Triage Agent · auto · 61.7 AIC · ⌖ 2.63 AIC · ⊞ 7.8K ·

@pelikhan
pelikhan marked this pull request as ready for review August 13, 2026 17:30
Copilot AI balanced review requested due to automatic review settings August 13, 2026 17:30

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Makes threat-detect installation failures non-fatal in warn mode while preserving strict and expression-configured behavior.

Changes:

  • Applies detection continue-on-error policy to binary installation.
  • Adds default warn-mode coverage.
  • Regenerates 104 affected workflows.
Show a summary per file
File Description
pkg/workflow/threat_detection_steps.go Applies failure policy to installation.
pkg/workflow/threat_detection_isolation_test.go Tests default warn mode.
.github/workflows/docs-noob-tester.lock.yml Regenerates tolerant install step.
.github/workflows/ci-coach.lock.yml Regenerates tolerant install step.
.github/workflows/commit-changes-analyzer.lock.yml Regenerates tolerant install step.
.github/workflows/ab-testing-advisor.lock.yml Regenerates tolerant install step.
.github/workflows/agentic-token-trend-audit.lock.yml Regenerates tolerant install step.
.github/workflows/copilot-session-insights.lock.yml Regenerates tolerant install step.
.github/workflows/breaking-change-checker.lock.yml Regenerates tolerant install step.
.github/workflows/daily-elixir-credo-snippet-audit.lock.yml Regenerates tolerant install step.
.github/workflows/daily-agentrx-trace-optimizer.lock.yml Regenerates tolerant install step.
.github/workflows/daily-cli-tools-tester.lock.yml Regenerates tolerant install step.
.github/workflows/blog-auditor.lock.yml Regenerates tolerant install step.
.github/workflows/cloclo.lock.yml Regenerates tolerant install step.
.github/workflows/q.lock.yml Regenerates tolerant install step.
.github/workflows/copilot-pr-prompt-analysis.lock.yml Regenerates tolerant install step.
.github/workflows/issue-monster.lock.yml Regenerates tolerant install step.
.github/workflows/daily-astrostylelite-markdown-spellcheck.lock.yml Regenerates tolerant install step.
.github/workflows/daily-code-metrics.lock.yml Regenerates tolerant install step.
.github/workflows/daily-multi-device-docs-tester.lock.yml Regenerates tolerant install step.
.github/workflows/daily-choice-test.lock.yml Regenerates tolerant install step.
.github/workflows/duplicate-code-detector.lock.yml Regenerates tolerant install step.
.github/workflows/daily-evals-report.lock.yml Regenerates tolerant install step.
.github/workflows/daily-byok-ollama-test.lock.yml Regenerates tolerant install step.
.github/workflows/daily-file-diet.lock.yml Regenerates tolerant install step.
.github/workflows/changeset.lock.yml Regenerates tolerant install step.
.github/workflows/daily-formal-spec-verifier.lock.yml Regenerates tolerant install step.
.github/workflows/test-quality-sentinel.lock.yml Regenerates tolerant install step.
.github/workflows/daily-hippo-learn.lock.yml Regenerates tolerant install step.
.github/workflows/daily-assign-issue-to-user.lock.yml Regenerates tolerant install step.
.github/workflows/copilot-pr-merged-report.lock.yml Regenerates tolerant install step.
.github/workflows/deep-report.lock.yml Regenerates tolerant install step.
.github/workflows/impeccable-skills-reviewer.lock.yml Regenerates tolerant install step.
.github/workflows/pr-code-quality-reviewer.lock.yml Regenerates tolerant install step.
.github/workflows/cli-version-checker.lock.yml Regenerates tolerant install step.
.github/workflows/github-remote-mcp-auth-test.lock.yml Regenerates tolerant install step.
.github/workflows/daily-doc-healer.lock.yml Regenerates tolerant install step.
.github/workflows/daily-experiment-report.lock.yml Regenerates tolerant install step.
.github/workflows/avenger.lock.yml Regenerates tolerant install step.
.github/workflows/copilot-opt.lock.yml Regenerates tolerant install step.
.github/workflows/daily-regulatory.lock.yml Regenerates tolerant install step.
.github/workflows/craft.lock.yml Regenerates tolerant install step.
.github/workflows/daily-model-inventory.lock.yml Regenerates tolerant install step.
.github/workflows/daily-news.lock.yml Regenerates tolerant install step.
.github/workflows/daily-performance-summary.lock.yml Regenerates tolerant install step.
.github/workflows/daily-reliability-review.lock.yml Regenerates tolerant install step.
.github/workflows/daily-doc-updater.lock.yml Regenerates tolerant install step.
.github/workflows/agent-performance-analyzer.lock.yml Regenerates tolerant install step.
.github/workflows/daily-token-consumption-report.lock.yml Regenerates tolerant install step.
.github/workflows/daily-mcp-concurrency-analysis.lock.yml Regenerates tolerant install step.
.github/workflows/audit-workflows.lock.yml Regenerates tolerant install step.
.github/workflows/api-consumption-report.lock.yml Regenerates tolerant install step.
.github/workflows/daily-cli-performance.lock.yml Regenerates tolerant install step.
.github/workflows/constraint-solving-potd.lock.yml Regenerates tolerant install step.
.github/workflows/daily-graft-intelligence.lock.yml Regenerates tolerant install step.
.github/workflows/chaos-pr-bundle-fuzzer.lock.yml Regenerates tolerant install step.
.github/workflows/ci-doctor.lock.yml Regenerates tolerant install step.
.github/workflows/contribution-check.lock.yml Regenerates tolerant install step.
.github/workflows/pr-sous-chef.lock.yml Regenerates tolerant install step.
.github/workflows/daily-compiler-quality.lock.yml Regenerates tolerant install step.
.github/workflows/design-decision-gate.lock.yml Regenerates tolerant install step.
.github/workflows/archivx-agentic-workflows-analyzer.lock.yml Regenerates tolerant install step.
.github/workflows/daily-function-namer.lock.yml Regenerates tolerant install step.
.github/workflows/daily-observability-report.lock.yml Regenerates tolerant install step.
.github/workflows/typist.lock.yml Regenerates tolerant install step.
.github/workflows/daily-ambient-context-optimizer.lock.yml Regenerates tolerant install step.
.github/workflows/prompt-clustering-analysis.lock.yml Regenerates tolerant install step.
.github/workflows/code-scanning-fixer.lock.yml Regenerates tolerant install step.
.github/workflows/aw-failure-investigator.lock.yml Regenerates tolerant install step.
.github/workflows/architecture-guardian.lock.yml Regenerates tolerant install step.
.github/workflows/github-mcp-structural-analysis.lock.yml Regenerates tolerant install step.
.github/workflows/daily-architecture-diagram.lock.yml Regenerates tolerant install step.
.github/workflows/copilot-agent-analysis.lock.yml Regenerates tolerant install step.
.github/workflows/daily-agent-of-the-day-blog-writer.lock.yml Regenerates tolerant install step.
.github/workflows/example-workflow-analyzer.lock.yml Regenerates tolerant install step.
.github/workflows/daily-credit-limit-test.lock.yml Regenerates tolerant install step.
.github/workflows/daily-compiler-threat-spec-optimizer.lock.yml Regenerates tolerant install step.
.github/workflows/pr-description-caveman.lock.yml Regenerates tolerant install step.
.github/workflows/deployment-incident-monitor.lock.yml Regenerates tolerant install step.
.github/workflows/eslint-monster.lock.yml Regenerates tolerant install step.
.github/workflows/copilot-cli-deep-research.lock.yml Regenerates tolerant install step.
.github/workflows/daily-awf-spec-compiler-surfacing.lock.yml Regenerates tolerant install step.
.github/workflows/ponytail-reviewer.lock.yml Regenerates tolerant install step.
.github/workflows/daily-cache-strategy-analyzer.lock.yml Regenerates tolerant install step.
.github/workflows/daily-max-ai-credits-test.lock.yml Regenerates tolerant install step.
.github/workflows/daily-caveman-optimizer.lock.yml Regenerates tolerant install step.
.github/workflows/daily-model-resolution.lock.yml Regenerates tolerant install step.
.github/workflows/cli-consistency-checker.lock.yml Regenerates tolerant install step.
.github/workflows/detection-analysis-report.lock.yml Regenerates tolerant install step.
.github/workflows/agent-job-health.lock.yml Regenerates tolerant install step.
.github/workflows/daily-issues-report.lock.yml Regenerates tolerant install step.
.github/workflows/artifacts-summary.lock.yml Regenerates tolerant install step.
.github/workflows/agent-persona-explorer.lock.yml Regenerates tolerant install step.
.github/workflows/daily-geo-optimizer.lock.yml Regenerates tolerant install step.
.github/workflows/approach-validator.lock.yml Regenerates tolerant install step.
.github/workflows/mattpocock-skills-reviewer.lock.yml Regenerates tolerant install step.
.github/workflows/archie.lock.yml Regenerates tolerant install step.
.github/workflows/daily-testify-uber-super-expert.lock.yml Regenerates tolerant install step.
.github/workflows/daily-aw-cross-repo-compile-check.lock.yml Regenerates tolerant install step.
.github/workflows/daily-community-attribution.lock.yml Regenerates tolerant install step.
.github/workflows/daily-repo-chronicle.lock.yml Regenerates tolerant install step.
.github/workflows/daily-rendering-scripts-verifier.lock.yml Regenerates tolerant install step.
.github/workflows/copilot-pr-nlp-analysis.lock.yml Regenerates tolerant install step.
.github/workflows/auto-triage-issues.lock.yml Regenerates tolerant install step.
.github/workflows/claude-code-user-docs-review.lock.yml Regenerates tolerant install step.
.github/workflows/daily-fact.lock.yml Regenerates tolerant install step.

Review details

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

  • Files reviewed: 106/106 changed files
  • Comments generated: 1
  • Review effort level: Balanced

Comment on lines +227 to +228
if !strings.Contains(installStepBlock, "continue-on-error: true") {
t.Error("Install threat-detect binary step must set continue-on-error: true in warn mode")

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added strict-mode and expression-mode test cases (TestExternalDetectorInstallStepContinueOnErrorStrictMode, TestExternalDetectorInstallStepContinueOnErrorExpressionMode) that assert the install step directly, in a248f03.

@github-actions

github-actions Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Ponytail Reviewer completed successfully!

Generated by Ponytail Reviewer for #52400

@github-actions

github-actions Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Design Decision Gate 🏗️ completed the design decision gate check.

No ADR enforcement needed: PR #52400 does not have the 'implementation' label and has only 44 new lines of code in business logic directories (threshold: 100).

🏗️ ADR gate enforced by Design Decision Gate 🏗️

@github-actions

github-actions Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

PR Code Quality Reviewer completed the code quality review.

🔎 Code quality review by PR Code Quality Reviewer

@github-actions

github-actions Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Test Quality Sentinel completed test quality analysis.

🧪 Test quality analysis by Test Quality Sentinel

@github-actions

github-actions Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes

This change weakens the strict-mode detection contract: the external detector install step now inherits the warn-mode continue-on-error behavior in every generated workflow, so a failed threat-detect download can be reported as success even when safe-outputs.threat-detection.continue-on-error: false is configured.

The blocking issue

The compiler already preserves strict vs warn behavior for the conclusion step, and buildInstallThreatDetectStep has tests/comments describing the same split. But the generated lockfiles in this PR all add continue-on-error: true unconditionally on the install step. That means a strict workflow cannot actually enforce binary availability anymore: the install step failure is swallowed before the detection job reaches the logic that is supposed to block safe outputs. Please add coverage for the strict and expression branches and make sure the recompiled workflows reflect the configured continue-on-error value instead of hard-coding true.

🔎 Code quality review by PR Code Quality Reviewer · gpt54 · 13.1 AIC · ⌖ 4.56 AIC · ⊞ 6.5K
Comment /review to run again

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ponytail review: one finding.

net: -8 lines possible.

Generated by ✂️ Ponytail Reviewer for #52400 · auto · 27.2 AIC · ⌖ 3.4 AIC · ⊞ 6.8K
Comment /ponytail to run again

// treat a failed/absent binary as a non-fatal detection failure via
// GH_AW_DETECTION_CONTINUE_ON_ERROR. Without continue-on-error on this install step, a
// transient download failure (e.g. a GitHub Releases CDN blip) would still mark this step —
// and therefore the whole detection job — as `failure`, even though the workflow logic

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

L519-524: yagni: third copy of the continue-on-error resolution block (data.SafeOutputs.ThreatDetection lookup). Extract a resolveContinueOnError(data) (bool, string) helper used by all three buildStep functions.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Extracted resolveThreatDetectionContinueOnError and reused it across all three build*Step functions in 4e7627b.

@github-actions github-actions Bot mentioned this pull request Aug 13, 2026
@github-actions

Copy link
Copy Markdown
Contributor

PR Triage

  • Category: chore
  • Risk: low
  • Priority: medium (score: 41/100)
  • Score breakdown: impact 12/50, urgency 15/30, quality 14/20
  • Recommended action: batch_review
  • Batch: chore-workflow-updates

Automated triage by PR Triage Agent.

Generated by 🔧 PR Triage Agent · auto · 64.3 AIC · ⌖ 2.94 AIC · ⊞ 7.8K ·

@github-actions

Copy link
Copy Markdown
Contributor

PR Triage

Category: chore  |  Risk: medium  |  Score: 33/100

Score breakdown

  • Impact: 15/50
  • Urgency: 10/30
  • Quality: 8/20

Recommended action: batch_review

Batch: chore-workflow-updates

Small logic change but regenerates 100 lock.yml files; no CI data captured.

Generated by 🔧 PR Triage Agent · auto · 68.2 AIC · ⌖ 10.7 AIC · ⊞ 7.8K ·

@pelikhan

Copy link
Copy Markdown
Collaborator

@copilot merge main and recompile

Copilot AI and others added 2 commits August 15, 2026 05:23
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI commented Aug 15, 2026

Copy link
Copy Markdown
Contributor Author

@copilot merge main and recompile

Merged main and recompiled all .lock.yml files. Also fixed a build break from the previous commit (missing resolveThreatDetectionContinueOnError helper) while doing so.

@gh-aw-bot

Copy link
Copy Markdown
Collaborator

@copilot Quick triage for maintainer-ready follow-up:

Please refresh the branch if needed, address the remaining maintainer-facing follow-up, and run the pr-finisher skill before handing this PR back to maintainers.

Outstanding review items (newest first):

  • github-actions: changes requested review remains open
  • github-actions: strict/expression continue-on-error behavior still needs confirmation after the latest merge-main refresh
  • copilot-pull-request-reviewer: verify the earlier review link feedback is fully addressed on the current branch

Failed checks from the compact candidate set:

  • None listed.

Branch update was requested automatically for this run when GitHub allows it.
Run context: https://github.com/github/gh-aw/actions/runs/31867777089

Generated by 👨‍🍳 PR Sous Chef · gpt54 · 33.2 AIC · ⌖ 5.85 AIC · ⊞ 8.7K ·
Comment /souschef to run again

…ntinue-on-error

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown
Contributor

PR Triage

Category: chore | Risk: medium | Score: 55/100

  • Impact: 20/50 (CI infra reliability, no runtime behavior change)
  • Urgency: 15/30 (mitigates transient CDN failures, not urgent)
  • Quality: 20/20 (clean retry/backoff fix, well-scoped)

Recommended action: defer
Large diff footprint (114 files) for a low-behavior-risk retry fix; batch with other workflow-metadata chores rather than fast-tracking.

Generated by 🔧 PR Triage Agent · auto · 46.6 AIC · ⌖ 2.63 AIC · ⊞ 8.1K ·

@gh-aw-bot

Copy link
Copy Markdown
Collaborator

@copilot Quick triage for maintainer-ready follow-up:

Please refresh the branch if needed, address the remaining maintainer-facing follow-up, and run the pr-finisher skill before handing this PR back to maintainers.

Outstanding review items (newest first):

  • github-actions: changes requested review remains open

Failed checks from the compact candidate set:

  • None listed.

Branch update was requested automatically for this run when GitHub allows it.
Run context: https://github.com/github/gh-aw/actions/runs/31871584565

Generated by 👨‍🍳 PR Sous Chef · gpt54 · 11.5 AIC · ⌖ 5.77 AIC · ⊞ 8.7K ·
Comment /souschef to run again

@gh-aw-bot

Copy link
Copy Markdown
Collaborator

@copilot Quick triage for maintainer-ready follow-up:

Please refresh the branch if needed, address any remaining maintainer-facing follow-up, and run the pr-finisher skill before handing this PR back to maintainers.

Outstanding review items (newest first):

  • No unresolved review threads were surfaced from available context; please verify current maintainer expectations before handoff.

Failed checks from the compact candidate set:

  • None listed.

Branch update was requested automatically for this run when GitHub allows it.
Run context: https://github.com/github/gh-aw/actions/runs/31874029539

Generated by 👨‍🍳 PR Sous Chef · gpt54 · 8.88 AIC · ⌖ 5.77 AIC · ⊞ 8.7K ·
Comment /souschef to run again

@github-actions

Copy link
Copy Markdown
Contributor

PR Triage

  • Category: chore
  • Risk: medium
  • Priority: medium
  • Score: 55/100
  • Recommended action: batch_review
  • Batch: workflow-metadata-chores

Adds continue-on-error tolerance to threat-detect installer across 104 recompiled lockfiles. CI passing. Note: existing labels on this PR are conflicting (multiple pr-priority/pr-action values) — recommend manual label cleanup.

Automated triage — see [PR Triage Report] for full context.

Generated by 🔧 PR Triage Agent · auto · 62.5 AIC · ⌖ 2.53 AIC · ⊞ 8.1K ·

@pelikhan
pelikhan merged commit c5fa87b into main Aug 15, 2026
2 checks passed
@pelikhan
pelikhan deleted the copilot/aw-failures-harden-binary-installers branch August 15, 2026 12:56
@github-actions

Copy link
Copy Markdown
Contributor

🎉 This pull request is included in a new release.

Release: v0.86.3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[aw-failures] [P0] Harden AWF/threat-detect binary installers against transient GitHub Releases connection failures

4 participants