Skip to content

fix: add explicit model for Copilot BYOK smoke test#2006

Merged
lpcox merged 7 commits intomainfrom
copilot/fix-byok-model-requirement
Apr 16, 2026
Merged

fix: add explicit model for Copilot BYOK smoke test#2006
lpcox merged 7 commits intomainfrom
copilot/fix-byok-model-requirement

Conversation

@lpcox
Copy link
Copy Markdown
Collaborator

@lpcox lpcox commented Apr 16, 2026

Problem

The Smoke Copilot BYOK workflow (run #24485448304) fails on all 4 attempts with:

BYOK providers require an explicit model. Run `copilot help providers` for configuration details.

Copilot CLI requires --model when running in BYOK/offline mode. The copilot_driver.cjs reads COPILOT_MODEL env var to pass --model, but the compiler's auto-generated step-level env sets it to ${{ vars.GH_AW_MODEL_AGENT_COPILOT || '' }} — which resolves to empty when the repo variable is unset.

Fix

  1. Set COPILOT_MODEL: gpt-4.1 in the workflow-level env: section
  2. Override the compiler's empty default fallback in the lock file to 'gpt-4.1'

This ensures the BYOK smoke test has a model even when the repo variable isn't configured.

Copilot CLI requires --model when running in BYOK/offline mode.
Without it, all attempts fail with: 'BYOK providers require an
explicit model.'

Set COPILOT_MODEL=gpt-4.1 in workflow env and override the
compiler's empty default fallback in the lock file.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@lpcox lpcox requested a review from Mossaka as a code owner April 16, 2026 00:41
Copilot AI review requested due to automatic review settings April 16, 2026 00:41
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 16, 2026

✅ Coverage Check Passed

Overall Coverage

Metric Base PR Delta
Lines 85.35% 85.43% 📈 +0.08%
Statements 85.24% 85.33% 📈 +0.09%
Functions 87.96% 87.96% ➡️ +0.00%
Branches 77.95% 78.00% 📈 +0.05%
📁 Per-file Coverage Changes (1 files)
File Lines (Before → After) Statements (Before → After)
src/docker-manager.ts 86.8% → 87.1% (+0.30%) 86.4% → 86.7% (+0.29%)

Coverage comparison generated by scripts/ci/compare-coverage.ts

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes the Copilot BYOK smoke workflow failing in offline/BYOK mode due to Copilot CLI requiring an explicit model when no repo variable provides one.

Changes:

  • Set a workflow-level default COPILOT_MODEL: gpt-4.1 for the BYOK smoke workflow source.
  • Update the compiled lock workflow to fall back to 'gpt-4.1' (instead of '') when vars.GH_AW_MODEL_AGENT_COPILOT is unset.
Show a summary per file
File Description
.github/workflows/smoke-copilot-byok.md Adds a workflow-level COPILOT_MODEL default so BYOK/offline runs always have an explicit model.
.github/workflows/smoke-copilot-byok.lock.yml Ensures the generated step env fallback doesn’t override the model with an empty string when the repo variable is unset.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

  • Files reviewed: 2/2 changed files
  • Comments generated: 0

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

api.githubcopilot.com requires the Copilot-Integration-Id header.
In BYOK mode, the Copilot CLI sends plain OpenAI-compatible
requests without Copilot-specific headers. The api-proxy sidecar
must inject this header when proxying to the Copilot API.

Uses 'copilot-developer-cli' as the default integration ID,
matching the Copilot CLI's own value. Configurable via
COPILOT_INTEGRATION_ID env var.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

- Add GitHub.com connectivity check and pre-fetched PR data comparison
- Add all 4 test sections: MCP, connectivity, file I/O, BYOK inference
- Update model from gpt-4.1 to gpt-5.4
- Recompile and post-process lock file

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

gpt-5.3-codex is not accessible via /chat/completions endpoint.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions
Copy link
Copy Markdown
Contributor

Smoke test results (run 24488563705)

✅ GitHub MCP — feat: add smoke test for Copilot CLI offline BYOK mode / fix(api-proxy): fix Gemini API_KEY_INVALID with credential isolation
✅ Playwright — Page title contains "GitHub"
✅ File write — /tmp/gh-aw/agent/smoke-test-claude-24488563705.txt created
✅ Bash verify — file read back successfully

Overall: PASS

💥 [THE END] — Illustrated by Smoke Claude

@github-actions
Copy link
Copy Markdown
Contributor

Smoke Test: Copilot BYOK (Offline) — ✅ PASS

Test Result
MCP PR list (#2003 feat: add smoke test for Copilot CLI offline BYOK mode)
GitHub.com connectivity
File write/read (smoke-test-copilot-byok-24488563695.txt)
BYOK inference (agent → api-proxy → api.githubcopilot.com)

Running in BYOK offline mode (COPILOT_OFFLINE=true) via api-proxy → api.githubcopilot.com.

cc @lpcox

🔑 BYOK report filed by Smoke Copilot BYOK

@github-actions
Copy link
Copy Markdown
Contributor

Smoke Test Results — OpenCode Engine

  • ✅ GitHub MCP: Read last 2 merged PRs (#2003 feat: add smoke test for Copilot CLI offline BYOK mode, #1995 fix(api-proxy): fix Gemini API_KEY_INVALID with credential isolation)
  • ✅ File write: /tmp/gh-aw/agent/smoke-test-opencode-24488563624.txt created and verified
  • ✅ Bash tool: file read back successfully
  • ✅ Build AWF: npm ci && npm run build succeeded

Overall: PASS

🌐 Transmitted by Smoke OpenCode

@github-actions
Copy link
Copy Markdown
Contributor

Smoke test (Codex) results:
PR titles: feat: add smoke test for Copilot CLI offline BYOK mode; fix(api-proxy): fix Gemini API_KEY_INVALID with credential isolation

  1. GitHub MCP ✅
  2. safeinputs-gh CLI ❌ (tool unavailable)
  3. Playwright title check ✅
  4. Tavily search ❌ (tool unavailable)
  5. File write ✅
  6. Bash cat verify ✅
  7. Discussion query/comment ❌ (required discussion tool unavailable)
  8. npm ci && npm run build ✅
    Overall status: FAIL

🔮 The oracle has spoken through Smoke Codex

@github-actions
Copy link
Copy Markdown
Contributor

🔥 AWF Smoke Test Results

PR: fix: add explicit model for Copilot BYOK smoke test — @lpcox

Test Result
GitHub API (merged PRs) ✅ PR #2003 retrieved
GitHub.com connectivity ✅ HTTP 200
File write/read ⚠️ Pre-step outputs not substituted (workflow template issue)

Overall: PASS (verifiable tests passed; pre-step data unavailable due to unsubstituted template vars)

📰 BREAKING: Report filed by Smoke Copilot

@github-actions
Copy link
Copy Markdown
Contributor

Smoke Test: GitHub Actions Services Connectivity ✅

Check Result
Redis PING (host.docker.internal:6379) PONG
PostgreSQL pg_isready (host.docker.internal:5432) ✅ accepting connections
PostgreSQL SELECT 1 (smoketest db, user postgres) ✅ returned 1

All checks passed. (redis-cli unavailable; Redis tested via Python socket — same protocol result.)

🔌 Service connectivity validated by Smoke Services

@github-actions
Copy link
Copy Markdown
Contributor

Chroot Version Comparison Results

Runtime Host Version Chroot Version Match?
Python Python 3.12.13 Python 3.12.3 ❌ NO
Node.js v24.14.1 v20.20.2 ❌ NO
Go go1.22.12 go1.22.12 ✅ YES

Overall: ❌ Not all tests passed — Python and Node.js versions differ between host and chroot environments.

Tested by Smoke Chroot

@github-actions
Copy link
Copy Markdown
Contributor

🏗️ Build Test Suite Results

Ecosystem Project Build/Install Tests Status
Bun elysia 1/1 passed ✅ PASS
Bun hono 1/1 passed ✅ PASS
C++ fmt N/A ✅ PASS
C++ json N/A ✅ PASS
Deno oak N/A 1/1 passed ✅ PASS
Deno std N/A 1/1 passed ✅ PASS
.NET hello-world N/A ✅ PASS
.NET json-parse N/A ✅ PASS
Go color 1/1 passed ✅ PASS
Go env 1/1 passed ✅ PASS
Go uuid 1/1 passed ✅ PASS
Java gson 1/1 passed ✅ PASS
Java caffeine 1/1 passed ✅ PASS
Node.js clsx All passed ✅ PASS
Node.js execa All passed ✅ PASS
Node.js p-limit All passed ✅ PASS
Rust fd 1/1 passed ✅ PASS
Rust zoxide 1/1 passed ✅ PASS

Overall: 8/8 ecosystems passed — ✅ PASS

Note: Java Maven required -Dmaven.repo.local=/tmp/gh-aw/agent/m2-repo workaround because /home/runner/.m2/ was owned by root (not writable). All tests passed successfully.

Generated by Build Test Suite for issue #2006 · ● 703.1K ·

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants