[copilot-cli-research] Copilot CLI Deep Research - 2026-03-24

[github-actions[bot]]

🔍 Overview

This is the first comprehensive Copilot CLI feature adoption analysis for this repository. Out of 177 total agentic workflows, 84 (47.5%) use the Copilot engine — making it by far the dominant engine. The analysis compared every available Copilot CLI configuration option against actual workflow usage to identify gaps, security opportunities, and optimizations.

Key finding: Several high-value Copilot CLI features have zero adoption despite being fully implemented and documented — including block-domains (network security), tools-startup-timeout (reliability), engine.env and engine.args (debugging/customization), and version pinning (reproducibility).

---

🔴 High Priority Issues

1. Zero usage of block-domains (security feature)
The block-domains network security feature has 0% adoption across all 177 workflows. This feature allows explicitly denying known-bad domains as an extra defense layer — it complements network.allowed by blocking specific subdomains or services.

2. 35.7% of Copilot workflows use unrestricted bash
30 of 84 Copilot workflows use bash: true, bash: ["*"], or bash: [":*"] — granting unrestricted shell access. This gives the AI agent write access to the entire runner filesystem without explicit tool-level scope.

3. copilot-requests feature at 49% adoption
Only 41 of 84 Copilot workflows use the copilot-requests: true feature, which uses the GitHub Actions token directly ($\{\{ github.token }}). The other 43 workflows still use the older COPILOT_GITHUB_TOKEN secret flow with extra auth overhead.

---

🟡 Medium Priority Opportunities

4. tools-startup-timeout: 0% usage
MCP server startup failures are a known flakiness source. The tools-startup-timeout field (available in frontmatter) allows customizing the MCP startup wait window — yet no workflow uses it. Workflows using Playwright or custom MCP servers are most vulnerable.

5. max-continuations: 1.2% usage (1 workflow)
Autopilot mode (--autopilot --max-autopilot-continues N) allows the agent to complete multiple consecutive passes — ideal for long-running research, refactoring, or batch operations. Only 1 workflow uses it. Several research/analysis workflows (e.g., agent-performance-analyzer, daily-copilot-token-report) would benefit.

6. GitHub toolset over-provisioning
Many workflows use toolsets: [default] (provides context, repos, issues, pull_requests, users) when they only need 1-2 toolsets. For example, auto-triage-issues.md correctly uses toolsets: [issues]. More workflows should follow this pattern to apply least-privilege.

---

📊 Feature Usage Matrix

Feature Category	Feature	Used	Out of	Usage %
Core Engine	copilot-requests: true	41	84	49%
Core Engine	strict: true (all workflows)	104	177	59%
Core Engine	engine.model	7	84	8%
Core Engine	engine.agent	15	84	18%
Core Engine	engine.args	0	84	0%
Core Engine	engine.env	0	84	0%
Core Engine	engine.command	0	84	0%
Core Engine	max-continuations	1	84	1.2%
Core Engine	engine.api-target	0	84	0%
Core Engine	Version pinning	0	84	0%
Sandbox/Network	AWF sandbox (agent: awf)	14	84	17%
Sandbox/Network	network: configured	85	177	48%
Sandbox/Network	block-domains:	0	177	0%
Tools	github: tool	~70	84	~83%
Tools	bash: (specific)	54	84	64%
Tools	bash: true/wildcard	30	84	36%
Tools	edit:	~40	84	~48%
Tools	web-fetch:	17	84	20%
Tools	web-search:	3	84	4%
Tools	playwright:	12	84	14%
Tools	serena:	6	84	7%
Tools	repo-memory:	~30	84	~36%
Tools	cache-memory:	52	177	29%
Tools	agentic-workflows:	26	177	15%
Tools	qmd: (vector search)	2	177	1.1%
Tools	mcp-scripts:	1	84	1.2%
GitHub Tool	toolsets: specified	~60	~70	~86%
GitHub Tool	allowed-repos:	11	177	6%
GitHub Tool	min-integrity:	11	177	6%
Reliability	tools-startup-timeout:	0	177	0%
Reliability	Version pinning	0	84	0%
Architecture	dispatch-workflow:	3	177	1.7%

---

3️⃣ Missed Opportunities (Detailed)

View High Priority Opportunities

🔴 Opportunity 1: Enable block-domains for Known-Bad Domains

• What: The block-domains field allows explicitly denying specific domains regardless of the network.allowed list — useful for blocking known data-exfiltration targets, malicious subdomains, or telemetry endpoints.
• Why It Matters: Defense-in-depth security. AWF firewall allows-by-default-deny is good, but adding explicit blocks catches edge cases and satisfies compliance requirements.
• Where: Any workflow with network: configured (85 workflows), especially those handling sensitive data (code-scanning-fixer.md, security-review.md).
• How:

  network:
    allowed:
      - defaults
      - github
    blocked:
      - "*.ngrok.io"         # block tunnel services
      - "*.requestbin.com"   # block request inspection

🔴 Opportunity 2: Reduce Unrestricted Bash Usage

• What: 30 workflows use bash: true, bash: ["*"], or bash: [":*"] — giving the AI unrestricted shell access. Many of these could use explicit command lists.
• Why It Matters: Smallest bash surface = smallest prompt-injection blast radius.
• Where: Workflows like daily-copilot-token-report.md (uses bash: ["*"]), craft.md (uses bash: ["*"]).
• How: Replace bash: ["*"] with explicit needed commands:

  # Instead of:
  tools:
    bash: ["*"]
  # Use:
  tools:
    bash:
      - "jq *"
      - "find .github -name '*.md'"
      - git
      - cat

🔴 Opportunity 3: Complete copilot-requests Adoption

• What: 43 of 84 Copilot workflows still use the legacy COPILOT_GITHUB_TOKEN secret instead of enabling copilot-requests: true (which uses $\{\{ github.token }} directly).
• Why It Matters: Simpler auth, no separate secret setup, uses native GitHub Actions token scoping.
• Where: Any Copilot workflow without features: copilot-requests: true.
• How: Add to frontmatter:

  features:
    copilot-requests: true

View Medium Priority Opportunities

🟡 Opportunity 4: Use tools-startup-timeout for MCP-Heavy Workflows

• What: MCP servers (Playwright, custom servers) sometimes take longer to start. tools-startup-timeout increases the wait window before the agent starts.
• Where: playwright: workflows (12 workflows), any workflow using Docker-based MCP servers.
• How:

  tools:
    playwright:
      version: "1.56.1"
  tools-startup-timeout: 120  # seconds (default is 60)

🟡 Opportunity 5: Leverage max-continuations for Research Workflows

• What: Only 1 workflow uses max-continuations. Research/analysis workflows that collect and synthesize data in multiple phases could benefit from autopilot mode.
• Where: agent-performance-analyzer.md, daily-copilot-token-report.md, copilot-pr-nlp-analysis.md.
• How:

  engine: copilot
  max-continuations: 3   # agent can do up to 3 consecutive runs

  Combined with cache-memory: to pass state between continuations.

🟡 Opportunity 6: Tighten GitHub Toolset Specificity

• What: Many workflows use toolsets: [default] or toolsets: [default, actions] when they only need specific API groups. default includes context, repos, issues, pull_requests, users.
• Available toolsets: context, repos, issues, pull_requests, users, actions, code_security, discussions, labels, notifications, orgs, projects, gists, search, dependabot, experiments, secret_protection, security_advisories, stargazers
• Where: Workflows that only read one or two types of data.
• Example fix for auto-triage-issues.md (already done correctly):

  tools:
    github:
      toolsets: [issues]  # not [default]

🟡 Opportunity 7: Pin Engine Versions for Stability

• What: Zero workflows pin Copilot CLI to a specific version. A CLI update can break any workflow without notice.
• Why: Reproducibility and stability — especially for customer-facing or scheduled workflows.
• How:

  engine:
    id: copilot
    version: "0.0.422"  # pin to a tested version

🟡 Opportunity 8: Expand min-integrity Usage

• What: Only 11 workflows use github.min-integrity. This filter prevents the agent from reading low-integrity content (e.g., content from anonymous users or untrusted forks) via GitHub tools.
• Where: Public-facing workflows that process user-submitted issues/PRs (auto-triage-issues.md, contribution-check.md, ai-moderator.md).
• How:

  tools:
    github:
      toolsets: [issues, pull_requests]
      min-integrity: approved

View Low Priority Opportunities

🟢 Opportunity 9: Use qmd for Documentation-Heavy Workflows

• What: Only 2 workflows use the qmd vector similarity search tool. Workflows that need to answer questions about codebase documentation could benefit from a local vector index.
• Where: daily-doc-updater.md, daily-doc-healer.md, cli-consistency-checker.md.
• How:

  tools:
    qmd:
      checkouts:
        - pattern: "docs/**/*.md"

🟢 Opportunity 10: Adopt mcp-scripts for Inline Tools

• What: Only 1 workflow uses mcp-scripts to define inline JavaScript/shell tools. These can encapsulate complex operations that would otherwise require full bash tool access.
• Where: Any workflow with complex data transformation needs.
• How:

  tools:
    mcp-scripts:
      analyze-metrics:
        script: |
          const data = await fs.readFile('/tmp/metrics.json');
          return JSON.parse(data).summary;
        description: "Parse and summarize metrics"

🟢 Opportunity 11: Use engine.env for Debug Configuration

• What: Zero workflows use engine.env for Copilot-specific environment variables. This can be used to pass debugging flags, custom endpoints, or feature toggles without hardcoding them.
• Use cases: COPILOT_DEBUG_TOOLS=1 for tool call logging, custom API routing, feature flags.
• How:

  engine:
    id: copilot
    env:
      COPILOT_AGENT_RUNNER_TYPE: STANDALONE  # already set by system
      CUSTOM_CONFIG: "value"

🟢 Opportunity 12: Leverage dispatch-workflow for Multi-Agent Orchestration

• What: Only 3 workflows use dispatch-workflow to fan out tasks to sub-agents. For complex meta-orchestration tasks, this enables parallel processing and specialization.
• Where: agent-performance-analyzer.md already creates sub-issues; it could instead dispatch specialized analysis agents.
• How:

  tools:
    dispatch-workflow:
      - id: code-analyzer
        workflow: code-analysis-agent

---

4️⃣ Specific Workflow Recommendations

View Workflow-Specific Recommendations

daily-copilot-token-report.md

• Current: bash: ["*"] (unrestricted), no max-continuations
• Recommended: Replace bash wildcard with explicit commands (jq, cat, find, sort); add max-continuations: 2 for multi-phase data collection + analysis

craft.md

• Current: bash: ["*"] (unrestricted)
• Recommended: Replace with specific commands (gh extension install, git, cat, find); the craft agent needs few commands beyond reading files and running gh aw compile

agent-performance-analyzer.md

• Current: repo-memory, no max-continuations
• Recommended: Add max-continuations: 3 to allow phase 1 (data collection) → phase 2 (analysis) → phase 3 (report writing); would significantly improve quality

code-scanning-fixer.md

• Current: Has github-token, specific toolsets, cache-memory — well configured
• Recommended: Add min-integrity: approved to github tool; add block-domains for extra security when handling security alerts

auto-triage-issues.md

• Current: Already uses toolsets: [issues] — excellent example of least-privilege
• Status: ✅ Best practice example — other workflows should follow this pattern

Workflows using AWF sandbox (14 workflows)

• Current: All use agent: awf correctly
• Recommended: Add block-domains to these workflows as an additional security layer since they handle sensitive operations

---

5️⃣ Trends & Historical Context

View Historical Trends

This is the first comprehensive analysis run (2026-03-24, run §23512121672).

Baseline established:

• 84/177 Copilot workflows (47.5%)
• 12 optimization opportunities identified
• Key zero-adoption features: block-domains, tools-startup-timeout, engine.env, engine.args, version pinning

Future analysis runs will track:

• Adoption of copilot-requests: true feature (target: 100%)
• Reduction in unrestricted bash (bash: ["*"]) usage
• Growth of max-continuations usage
• block-domains adoption
• GitHub toolset specificity improvements

---

6️⃣ Best Practice Guidelines

Based on this analysis, here are the top 5 best practices for Copilot workflows:

1. Minimal bash access: Use bash: [specific-commands] instead of bash: true or bash: ["*"]. Reference auto-triage-issues.md (bash: ["jq *"]) as a model.

2. Enable copilot-requests: true: All new Copilot workflows should include features: copilot-requests: true to use the simpler, native GitHub Actions token flow.

3. Specific GitHub toolsets: Never use the full default toolset unless all 5 toolset groups are actually needed. Start with the minimum (e.g., toolsets: [issues]) and add as required.

4. Use AWF sandbox + block-domains together: For any workflow that handles untrusted input (issues, PRs, comments), combine network: allowed: [defaults] with block-domains to apply defense-in-depth.

5. Add tools-startup-timeout for MCP workflows: Any workflow with Playwright, custom MCP servers, or Docker-based tools should set tools-startup-timeout: 90 or higher to reduce flakiness.

---

7️⃣ Action Items

Immediate (this week):

• Add features: copilot-requests: true to the 43 Copilot workflows missing it
• Replace bash: ["*"] with explicit command lists in craft.md and daily-copilot-token-report.md
• Add block-domains to workflows handling security-sensitive data

Short-term (this month):

• Audit and tighten GitHub toolset specificity across 30+ workflows using [default]
• Add max-continuations: 2-3 to multi-phase research workflows
• Add tools-startup-timeout: 90 to all Playwright-using workflows
• Add min-integrity: approved to public-facing issue/PR processing workflows

Long-term (this quarter):

• Pin Copilot CLI version in critical/scheduled workflows for reproducibility
• Expand qmd adoption for documentation-heavy analysis workflows
• Build mcp-scripts examples in shared imports for common data-transform patterns
• Schedule monthly re-runs of this workflow to track adoption trends

---

View Supporting Evidence & Methodology

Research Methodology

Phase 1 - Feature Inventory: Examined pkg/workflow/copilot_engine.go, copilot_engine_execution.go, copilot_engine_tools.go, pkg/workflow/engine.go, and docs/src/content/docs/reference/engines.md to build a complete list of available Copilot CLI configuration options.

Phase 2 - Usage Analysis: Used grep pattern matching across all 177 workflow markdown files to count feature adoption. Examined sample workflows (archie.md, craft.md, auto-triage-issues.md, code-scanning-fixer.md, agent-performance-analyzer.md) for qualitative pattern analysis.

Phase 3 - Gap Analysis: Compared available features against measured usage to identify zero-adoption and low-adoption features.

Data Sources:

• Go source: pkg/workflow/copilot_engine*.go, pkg/workflow/engine.go, pkg/constants/constants.go
• Documentation: docs/src/content/docs/reference/engines.md, docs/src/content/docs/reference/tools.md, docs/src/content/docs/reference/github-tools.md
• Workflows: .github/workflows/*.md (177 files, 84 using Copilot)
• CHANGELOG: Recent feature additions

Repo Memory: This analysis was intended to be saved to memory/copilot-cli-research/ on the memory/copilot-cli-research branch for trend tracking. The branch is initialized; future runs will persist data for longitudinal comparison.

---

References:

• §23512121672 — This workflow run
• Copilot Engine Docs — Complete engine reference
• GitHub Tools Docs — GitHub toolsets and options

AI generated by Copilot CLI Deep Research Agent · history

• expires on Mar 25, 2026, 9:14 PM UTC

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-03-25T21:14:53.011Z.

Closed by Workflow