⚡ Copilot Token Optimization2026-09-19 — Smoke Copilot BYOK AOAI (Entra)

[github-actions]

Target Workflow: Smoke Copilot BYOK AOAI (Entra)

Source report: #8348 (also flagged in the 2026-09-06 and 2026-09-08 reports)
Estimated cost per run: $0.00 (cost tracking not populated; token volume is the actionable signal)
Total tokens per run: ~185K avg (peaked at 233K in the 2026-09-06 report, 167K in 2026-09-08)
Cache hit rate: not reported (no token_usage_summary cache breakdown available in the source data)
LLM turns: not reported by the analyzer (Turns column shows 0 for all workflows — instrumentation gap), but working_set.invocations on comparable smoke/analyzer runs in the last 7 days shows 8–11 context rebuilds per run with rebuild_factor ~8.7x, which is the dominant cost driver for small-prompt / multi-turn agentic workflows like this one.

This workflow is the highest average-token workflow with an open slot in the latest report (Smoke Copilot BYOK AOAI (api-key) and Duplicate Code Detector are higher/tied in some windows, but Duplicate Code Detector already has an open optimization issue).

Current Configuration

Setting	Value
Tools loaded	bash: ["*"] (unrestricted), github: {toolsets: [pull_requests]}
Tools actually used	github-list_pull_requests (1 call, optional/best-effort), bash cat (1 file existence check)
Network groups	defaults, github, login.microsoftonline.com (all in active use — no trimming needed)
Pre-agent steps	Yes — steps: pre-computes PR data, HTTP check, and file write/read, writing results to /tmp/gh-aw/agent/smoke-context.txt
Post-agent steps	Yes — post-steps: validates safe-output invocation and greps firewall logs for Azure OpenAI traffic
Prompt size	~4,183 chars (~1,045 tokens) — the markdown body itself is small and not the problem
Recent run reliability	Both of the last 2 runs (schedule + PR) ended in driver_exit failure with token_usage: 0 recorded — failures are silently inflating the reported per-run average from prior successful runs and wasting the full Action minutes budget on retries

Recommendations

1. Drop the live github: MCP tool and rely solely on the pre-fetched PR data

Estimated savings: ~8–12K tokens per turn × up to ~10 context rebuilds/run ≈ 80–120K tokens/run (~45–65%)

The prompt already has full fallback logic: "If the tool is unavailable or its response is filtered by secrecy policy, validate the Pre-Fetched PR Data instead... Do not call missing_tool." This means the live MCP call is optional and only used as a redundant check against data the pre-agent steps: block already fetched via gh pr list. Loading the github MCP toolset (even restricted to pull_requests) still injects tool schemas (~8-12K tokens) into every turn of the conversation, and with an OIDC/Azure auth flow already flagged as flaky (2/2 recent driver_exit failures), the tool surface is pure overhead most runs never need.

Change: Remove the github: block from tools: entirely in .github/workflows/smoke-copilot-byok-aoai-entra.md:

tools:
  bash:
    - "cat"
    - "echo"
# github: removed — pre-fetched PR data in /tmp/gh-aw/agent/smoke-context.txt is sufficient

Update the prompt's "GitHub MCP Testing" section to drop the live-call branch and always validate against smoke-pr-data.txt, since that's already the documented fallback behavior.

2. Restrict bash: ["*"] to the specific commands the prompt actually needs

Estimated savings: ~1–3K tokens/turn (~1-2%)

The agent's only bash usage per the prompt is cat on the pre-computed test file (test #3). A wildcard bash: ["*"] forces the full unrestricted-bash tool description to be loaded. Scoping it to the literal commands used reduces tool-schema token cost and shrinks the sandboxed command surface (defense in depth).

tools:
  bash:
    - "cat"

3. Fix the underlying driver_exit failures before the next optimization pass

Estimated savings: Indirect — eliminates ~5 wasted Action-minutes and a full context-rebuild cycle per failed run (2/2 recent runs failed)

Both of the last two runs (35410858061 schedule run, 35402969445 PR run) ended in driver_exit with error_count: 1 and token_usage: 0 — meaning the analyzer can't even attribute cost to these runs, but the full container/agent lifecycle (and associated OIDC → Azure AD token exchange) still executes and burns real minutes every 12h on schedule. Since this workflow already appeared in two prior reports at 167K–233K tokens/run when it was succeeding, stabilizing the OIDC exchange (or the driver) is a prerequisite to trusting future token measurements for this workflow and to stop the schedule from silently repeating a failing path every 12 hours.

4. Reduce turn count by eliminating the optional/branching verification step

Estimated savings: ~1 turn/run, difficult to quantify precisely without turn telemetry, but removing branch logic ("try live MCP call, else fall back") collapses two possible code paths into one, which the analyzer's own report currently can't measure (Turns column shows 0 across all workflows — worth flagging as an instrumentation gap in the analyzer, not just this workflow)

Since recommendation #1 removes the live MCP call, the "if unavailable, fall back" branching in the prompt becomes dead logic and can be deleted, simplifying the agent's single-pass execution to: read context file → confirm HTTP code → cat test file → confirm inference is live → call add_comment/noop. This is a straight-line task requiring no tool-choice reasoning, which should also reduce reasoning/output tokens.

Expected Impact

Metric	Current	Projected	Savings
Total tokens/run	~185K (avg across recent reports)	~65–100K	-45% to -65%
Cost/run	$0.00 (untracked)	$0.00 (untracked)	n/a — track via awf logs stats post-change
LLM turns	Unmeasured (0 reported)	Fewer (single straight-line pass)	Qualitative
Session time	~4.9m (recent runs, both failed)	Est. same or lower once driver_exit is fixed	-X% pending fix

Implementation Checklist

• Remove github: toolset from tools: in .github/workflows/smoke-copilot-byok-aoai-entra.md; update the "GitHub MCP Testing" prompt section to validate only against pre-fetched data
• Restrict bash: from ["*"] to ["cat"] (only command actually used by the agent)
• Investigate and fix the driver_exit root cause behind the last 2 failed runs (35410858061, 35402969445) before re-measuring token usage
• Simplify the prompt's Output section to remove now-dead "live MCP unavailable" fallback branching
• Recompile: gh aw compile .github/workflows/smoke-copilot-byok-aoai-entra.md
• Post-process: npx tsx scripts/ci/postprocess-smoke-workflows.ts
• Verify CI passes on PR
• Compare token usage on new run vs baseline (185K avg) via awf logs stats or the next copilot-token-usage-analyzer report

Generated by Daily Copilot Token Optimization Advisor · copilot · auto · 103.9 AIC · ⊞ 9.8K · ◷

[github-actions]

This issue is being closed as outdated. A newer issue has been created: #8794

View newer issue

---

This action was performed automatically by the Daily Copilot Token Optimization Advisor workflow.