Motivation
Link to the gh-aw PR: github/gh-aw#63474 — Enforce firewall compatibility for Copilot and web tools; validation rejects (strict mode) or warns (non-strict) when Copilot's built-in web-fetch/web-search tools are combined with restricted network.allowed, since these tools bypass the firewall boundary. github/gh-aw#63632 later scoped this validation to only fire when Copilot actually enables web-fetch or web-search.
Proposed test
- Workflow file:
test-copilot-network-strict-web-tools-rejection.md
- Trigger:
workflow_dispatch
- Engine: copilot
- Safe output: none required — this is a compile-time validation test
- Variant: standard
Minimal test prompt sketch
Author a workflow with engine: copilot, network: { allowed: [defaults] } (a restricted policy), and Copilot's built-in web-fetch tool enabled, using strict: true. Assert via gh aw compile that compilation fails with a diagnostic identifying the firewall/network-tool incompatibility. A companion non-strict variant (or the same file with strict: false) should compile successfully but emit a warning.
New fixtures or secrets needed
None — this only exercises the compiler; no live engine call or safe-output needed. Could be validated with a small script step in CI rather than the full e2e harness if workflow_dispatch execution assertions are awkward for compile-failure cases (see e2e.sh's existing COMPILE_FAILED_FILE handling for tests expected not to compile).
Notes
Not covered by any existing test-copilot-network-* workflow (network-isolation, network-engine-domain-opt-in), which only test the network-allowed-domain happy path, not the strict-mode Copilot+web-tool rejection path. Not present in fails.txt or open suggestions.
Generated by 🔍 Suggest New E2E Tests · copilot · auto · 65.1 AIC · ⌖ 9.43 AIC · ⊞ 8.5K · ◷
Motivation
Link to the gh-aw PR: github/gh-aw#63474 — Enforce firewall compatibility for Copilot and web tools; validation rejects (strict mode) or warns (non-strict) when Copilot's built-in
web-fetch/web-searchtools are combined with restrictednetwork.allowed, since these tools bypass the firewall boundary. github/gh-aw#63632 later scoped this validation to only fire when Copilot actually enablesweb-fetchorweb-search.Proposed test
test-copilot-network-strict-web-tools-rejection.mdworkflow_dispatchMinimal test prompt sketch
Author a workflow with
engine: copilot,network: { allowed: [defaults] }(a restricted policy), and Copilot's built-inweb-fetchtool enabled, usingstrict: true. Assert viagh aw compilethat compilation fails with a diagnostic identifying the firewall/network-tool incompatibility. A companion non-strict variant (or the same file withstrict: false) should compile successfully but emit a warning.New fixtures or secrets needed
None — this only exercises the compiler; no live engine call or safe-output needed. Could be validated with a small script step in CI rather than the full e2e harness if
workflow_dispatchexecution assertions are awkward for compile-failure cases (seee2e.sh's existingCOMPILE_FAILED_FILEhandling for tests expected not to compile).Notes
Not covered by any existing
test-copilot-network-*workflow (network-isolation,network-engine-domain-opt-in), which only test the network-allowed-domain happy path, not the strict-mode Copilot+web-tool rejection path. Not present infails.txtor open suggestions.