fix: pin smoke-claude to claude-sonnet-4.6 to resolve model_policy_violation 403 by Copilot · Pull Request #46971 · github/gh-aw

Copilot · 2026-07-21T04:45:46Z

smoke-claude was deterministically failing because the Claude CLI defaults to claude-opus-4-8 when no model is configured, and the Anthropic API key's model policy explicitly disallows that model — producing a non-retryable 403 model_policy_violation on the first API call (0 turns, 0 tokens every run).

Changes

smoke-claude.md — adds model: claude-sonnet-4.6 under engine: to pin a policy-permitted model:

engine:
  id: claude
  model: claude-sonnet-4.6   # ← added
  bare: true

smoke-claude.lock.yml — recompiled; ANTHROPIC_MODEL: claude-sonnet-4.6 now injected into the Execute Claude step, overriding the CLI default.

The workflow already had models: blocked: ["*opus*"] as a proxy-level guard, but the root fix is to never let the CLI default to an opus model in the first place.

Fixes [aw-failures] [aw-fix] P1: Smoke Claude hard-fails — claude-opus-4-8 blocked by model policy (403 model_policy_violation) #46845

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

pi.dev

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "pi.dev"

See Network Configuration for more information.

Generated by 👨‍🍳 PR Sous Chef · 20.6 AIC · ⌖ 6.84 AIC · ⊞ 7K · ◷
_{Comment /souschef to run again}

✨ PR Review Safe Output Test - Run 29804334966

💥 [THE END] — Illustrated by Smoke Claude · 80.9 AIC · ⌖ 19.4 AIC · ⊞ 6.1K · ◷
_{Comment /smoke-claude to run again}

…ation Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

github-actions · 2026-07-21T05:00:33Z

✅ PR Code Quality Reviewer completed the code quality review.

github-actions · 2026-07-21T05:00:42Z

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

github-actions · 2026-07-21T05:01:08Z

✅ Test Quality Sentinel completed test quality analysis.

No test files were added or modified in this PR. Test Quality Sentinel skipped.

github-actions · 2026-07-21T05:01:42Z

✅ Design Decision Gate 🏗️ completed the design decision gate check.

No ADR enforcement needed: PR does not have the 'implementation' label and has ≤100 new lines of code in business logic directories (0 additions detected).

Copilot

Pull request overview

Pins Smoke Claude to Sonnet 4.6 to avoid the disallowed Opus default, but the configured model ID is malformed.

Changes:

Adds an explicit Claude model.
Regenerates the compiled workflow with that model.

Show a summary per file

File	Description
`.github/workflows/smoke-claude.md`	Configures the smoke workflow model.
`.github/workflows/smoke-claude.lock.yml`	Propagates the model into generated runtime configuration.

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Files reviewed: 2/2 changed files
Comments generated: 1
Review effort level: Medium

 max-turns: 100
 engine:
  id: claude
+  model: claude-sonnet-4.6


github-actions

The fix is correct and minimal. Pinning ANTHROPIC_MODEL=claude-sonnet-4.6 via the env var and removing the dynamic --model flag from both the agent and detection job invocations is the right approach to avoid the model_policy_violation 403. Lock file metadata is consistently updated. No issues found.

🧵 Reviewed using Impeccable skills by Impeccable Skills Reviewer · 11.7 AIC · ⌖ 4.33 AIC · ⊞ 5K

github-actions

Skills-Based Review 🧠

Applied /diagnosing-bugs — the root cause is correctly identified and addressed.

📋 Assessment

Root cause analysis ✅

The PR correctly identifies that the Claude CLI defaults to claude-opus-4-8 when no model is configured, and that model is policy-blocked. The fix pins claude-sonnet-4.6 in three consistent places:

smoke-claude.md engine config (source of truth)
ANTHROPIC_MODEL env var (Claude CLI picks this up natively)
GH_AW_ENGINE_MODEL / GH_AW_INFO_MODEL hardcoded (telemetry/reporting)

Regression coverage ✅

The smoke workflow itself is the regression test — a successful run proves the fix holds.

Positive highlights

✅ Both agent and detection phases are fixed consistently
✅ The variable-based override (GH_AW_MODEL_AGENT_CLAUDE) that was masking the bug is removed cleanly
✅ agent_model is now captured in the lock file metadata, improving auditability
✅ PR description is clear and thorough about the failure mode

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · 17.4 AIC · ⌖ 4.44 AIC · ⊞ 6.7K
_{Comment /matt to run again}

github-actions

Review: Non-blocking observations

The fix correctly addresses the root cause — pinning an explicit permitted model to avoid the 403 model_policy_violation. Two maintainability concerns worth addressing in a follow-up:

Issues found

Runtime override removed (smoke-claude.md:78): Removing the GH_AW_MODEL_AGENT_CLAUDE variable fallback takes away the operator escape hatch for model switching without a code change. A variable-with-default pattern would be safer.
GH_AW_ENGINE_MODEL hardcoded (smoke-claude.lock.yml:65): Previously reflected the actual model from job output; now it's a static string that can silently diverge from reality if model selection ever changes.

Neither is blocking for the immediate fix, but both reduce operational flexibility and telemetry reliability.

🔎 Code quality review by PR Code Quality Reviewer · 32.5 AIC · ⌖ 4.55 AIC · ⊞ 5.6K
_{Comment /review to run again}

Comments that could not be inline-anchored

.github/workflows/smoke-claude.md:78

Loss of runtime model override: Hardcoding the model removes the ability to swap models via repo/org variables without a code commit.

Previously the agent phase used ${GH_AW_MODEL_AGENT_CLAUDE:+ --model "$GH_AW_MODEL_AGENT_CLAUDE"} so operators could override the model by setting a repo/org variable — now that escape hatch is gone. If Anthropic retires claude-sonnet-4.6 or the team needs to test an alternative, a code change and recompile is required instead of a variable toggle. Consi…

.github/workflows/smoke-claude.lock.yml:65

GH_AW_ENGINE_MODEL hardcoded breaks telemetry accuracy: Previously sourced from needs.agent.outputs.model (the actual model used), now hardcoded to "claude-sonnet-4.6" independently. If the agent job ever runs with a different model (env var ignored, binary fallback, future override), this field silently reports the wrong model in all downstream telemetry and observability pipelines. The old dynamic approach was more correct.

pelikhan · 2026-07-21T05:06:06Z

/smoke-claude

github-actions · 2026-07-21T05:06:37Z

💫 TO BE CONTINUED... Smoke Claude failed! Our hero faces unexpected challenges...

gh-aw-bot · 2026-07-21T05:22:08Z

@copilot please run the pr-finisher skill, review any remaining unresolved feedback , rerun checks once the branch is up to date

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

pi.dev

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "pi.dev"

See Network Configuration for more information.

Generated by 👨‍🍳 PR Sous Chef · 20.6 AIC · ⌖ 6.84 AIC · ⊞ 7K · ◷
_{Comment /souschef to run again}

pelikhan · 2026-07-21T05:23:23Z

@copilot Claude failed https://github.com/github/gh-aw/actions/runs/29803065321/job/88547910020#step:46:1

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

Copilot · 2026-07-21T05:31:04Z

/smoke-claude

Fixed in commit fix: use correct model ID claude-sonnet-4-6 (hyphens not dots). The model ID is now claude-sonnet-4-6 in both smoke-claude.md and the recompiled smoke-claude.lock.yml.

pelikhan · 2026-07-21T05:33:01Z

/smoke-claude

github-actions · 2026-07-21T05:33:29Z

🎬 THE END — Smoke Claude MISSION: ACCOMPLISHED! The hero saves the day! ✨

Copilot · 2026-07-21T05:38:01Z

@copilot Claude failed https://github.com/github/gh-aw/actions/runs/29803065321/job/88547910020#step:46:1

That run was on commit 46b85df which had the incorrect model ID claude-sonnet-4.6 (dots). The fix is already pushed in commit de79b66 which changes it to claude-sonnet-4-6 (hyphens). The workflow needs to be re-run on the latest commit to pick up the corrected model ID.

github-actions · 2026-07-21T05:45:51Z

Smoke Test Run 29804334966 — Quick Results

Core tests #1-12: ✅✅✅❌✅✅✅✅✅✅✅✅
PR review tests #13-19: ✅✅✅✅✅✅⚠️

Overall: PARTIAL (Playwright ❌ — Chrome daemon crashes in sandbox)

See issue #46983 for full details.

💥 [THE END] — Illustrated by Smoke Claude · 80.9 AIC · ⌖ 19.4 AIC · ⊞ 6.1K · ◷
_{Comment /smoke-claude to run again}

github-actions

💥 Automated smoke test review - all systems nominal! The model pinning fix looks correct and well-implemented. — Run 29804334966

💥 [THE END] — Illustrated by Smoke Claude · 80.9 AIC · ⌖ 19.4 AIC · ⊞ 6.1K
_{Comment /smoke-claude to run again}

github-actions · 2026-07-21T05:46:00Z

          GH_AW_INFO_ENGINE_ID: "claude"
          GH_AW_INFO_ENGINE_NAME: "Claude Code"
-          GH_AW_INFO_MODEL: ${{ vars.GH_AW_MODEL_AGENT_CLAUDE || vars.GH_AW_DEFAULT_MODEL_CLAUDE || 'agent' }}
+          GH_AW_INFO_MODEL: "claude-sonnet-4-6"


✅ Good fix: hardcoding claude-sonnet-4-6 as the model directly in the workflow ensures the correct model is always used, regardless of repo variables being set or not. This is a more reliable approach than depending on vars.GH_AW_MODEL_AGENT_CLAUDE.

github-actions · 2026-07-21T05:46:00Z

 max-turns: 100
 engine:
  id: claude
+  model: claude-sonnet-4-6


💡 Pinning the model in the workflow frontmatter is the right approach — it propagates to the compiled lock file via ANTHROPIC_MODEL env var, ensuring consistent model usage across all agent invocations. Consider also documenting why this specific model is chosen in a comment.

github-actions · 2026-07-21T05:52:35Z

Thanks for the fix! 🎯 This focused workflow config update resolves the deterministic Claude model policy violation by pinning to claude-sonnet-4.6. The change is well-described and directly addresses issue #46845. The fix is clean and aligned with the gh-aw contribution guidelines.

No further action needed — this looks ready for review! ✅

Generated by ✅ Contribution Check · 68.1 AIC · ⌖ 9.02 AIC · ⊞ 6.2K · ◷

github-actions · 2026-07-21T08:27:44Z

🎉 This pull request is included in a new release.

Release: v0.82.15

Initial plan

58cac5d

Copilot AI assigned Copilot and pelikhan Jul 21, 2026

Copilot started work on behalf of pelikhan July 21, 2026 04:46 View session

Copilot AI linked an issue Jul 21, 2026 that may be closed by this pull request

[aw-failures] [aw-fix] P1: Smoke Claude hard-fails — claude-opus-4-8 blocked by model policy (403 model_policy_violation) #46845

Closed

fix: pin smoke-claude to claude-sonnet-4.6 to avoid model_policy_viol…

46b85df

…ation Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Repoint Smoke Claude engine from claude-opus-4-8~~ fix: pin smoke-claude to claude-sonnet-4.6 to resolve model_policy_violation 403 Jul 21, 2026

Copilot finished work on behalf of pelikhan July 21, 2026 04:59

Copilot AI requested a review from pelikhan July 21, 2026 04:59

pelikhan marked this pull request as ready for review July 21, 2026 04:59

Copilot AI review requested due to automatic review settings July 21, 2026 04:59

Copilot started reviewing on behalf of pelikhan July 21, 2026 05:00 View session

Copilot AI reviewed Jul 21, 2026

View reviewed changes

Comment thread .github/workflows/smoke-claude.md Outdated

max-turns: 100

engine:

id: claude

model: claude-sonnet-4.6

github-actions Bot approved these changes Jul 21, 2026

View reviewed changes

github-actions Bot mentioned this pull request Jul 21, 2026

[aw] Matt Pocock Skills Reviewer failed #46980

Closed

github-actions Bot reviewed Jul 21, 2026

View reviewed changes

github-actions Bot mentioned this pull request Jul 21, 2026

[pr-sous-chef] PR Sous Chef run report #46937

Closed

Copilot started work on behalf of gh-aw-bot July 21, 2026 05:22 View session

fix: use correct model ID claude-sonnet-4-6 (hyphens not dots)

de79b66

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

Copilot finished work on behalf of gh-aw-bot July 21, 2026 05:31

Copilot AI requested a review from gh-aw-bot July 21, 2026 05:31

Copilot started work on behalf of pelikhan July 21, 2026 05:31 View session

github-actions Bot mentioned this pull request Jul 21, 2026

Smoke Test: Claude - 29804334966 #46983

Closed

github-actions Bot added the smoke-claude label Jul 21, 2026

github-actions Bot reviewed Jul 21, 2026

View reviewed changes

Copilot finished work on behalf of pelikhan July 21, 2026 05:46

github-actions Bot mentioned this pull request Jul 21, 2026

[Contribution Check Report] Contribution Check — 2026-07-21 #46947

Closed

pelikhan merged commit 9a6af64 into main Jul 21, 2026
29 checks passed

pelikhan deleted the copilot/aw-failures-fix-smoke-claude branch July 21, 2026 05:52

github-actions Bot mentioned this pull request Jul 21, 2026

Smoke Test: MAI-Code-1-Flash - 29812424728 #47025

Closed

Uh oh!

Conversation

Copilot AI commented Jul 21, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Uh oh!

github-actions Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Review details

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Skills-Based Review 🧠

Root cause analysis ✅

Regression coverage ✅

Positive highlights

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Review: Non-blocking observations

Comments that could not be inline-anchored

Uh oh!

pelikhan commented Jul 21, 2026

Uh oh!

github-actions Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gh-aw-bot commented Jul 21, 2026

Uh oh!

pelikhan commented Jul 21, 2026

Uh oh!

Copilot AI commented Jul 21, 2026

Uh oh!

pelikhan commented Jul 21, 2026

Uh oh!

github-actions Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI commented Jul 21, 2026

Uh oh!

github-actions Bot commented Jul 21, 2026

Smoke Test Run 29804334966 — Quick Results

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 21, 2026

Uh oh!

Uh oh!

github-actions Bot commented Jul 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

Copilot AI commented Jul 21, 2026 •

edited by github-actions Bot

Loading

github-actions Bot commented Jul 21, 2026 •

edited

Loading

github-actions Bot commented Jul 21, 2026 •

edited

Loading

github-actions Bot commented Jul 21, 2026 •

edited

Loading

github-actions Bot commented Jul 21, 2026 •

edited

Loading

github-actions Bot commented Jul 21, 2026 •

edited

Loading

github-actions Bot commented Jul 21, 2026 •

edited

Loading