From 13ba8092e27bb9decee515b21ac6ca35dc88a4b5 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Wed, 15 Jul 2026 21:58:20 +0000 Subject: [PATCH 1/4] Create .github/aw/evals.md with BinEval evals syntax and methodology guide Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> --- .github/aw/evals.md | 221 ++++++++++++++++++ .github/aw/syntax.md | 1 + .github/skills/agentic-workflows/SKILL.md | 1 + .../agentic_workflows_fallback_aw_files.json | 1 + 4 files changed, 224 insertions(+) create mode 100644 .github/aw/evals.md diff --git a/.github/aw/evals.md b/.github/aw/evals.md new file mode 100644 index 00000000000..1a3ddf84aaf --- /dev/null +++ b/.github/aw/evals.md @@ -0,0 +1,221 @@ +--- +description: Guide for adding BinEval-style binary evaluations to agentic workflows — syntax, question decomposition methodology, result storage, and anti-patterns. +--- + +# BinEval Evaluations in Agentic Workflows + +Evals let you verify automatically whether an agentic run met its goals. Each evaluation is a binary YES/NO question answered by an LLM judge that reads the agent's output. Results are stored as `evals.jsonl` artifacts and persisted to a dedicated git branch for historical comparison. + +--- + +## How Evals Work + +Per run: + +1. **Setup** — the evals job downloads the agent artifact (`agent_output.json`, `prompt.txt`) and writes a BinEval prompt containing all declared questions. +2. **Execute** — an LLM judge runs in a network-restricted sandbox (same engine as the agent job) and answers each question with YES or NO. +3. **Parse** — raw engine output is parsed into per-question records and written to `evals.jsonl`. +4. **Redact** — any credential patterns are removed from the results before upload. +5. **Upload** — `evals.jsonl` is uploaded as the `evals` workflow artifact and optionally persisted to the `evals/` git branch. + +The evals job runs **after** the agent job and **in parallel with** `safe_outputs`, so it does not block the write path. + +--- + +## Basic Syntax + +### Shorthand — plain list + +```yaml +--- +on: + issues: + types: [opened] +engine: copilot +evals: + - id: scoped_change + question: Is the implementation limited to the scope described in the issue? + - id: no_regressions + question: Does the change avoid modifying files unrelated to the task? +--- + +Implement the requested change described in ${{ github.event.issue.body }}. +``` + +Each entry requires: + +- `id` — unique identifier for the question (used as the key in `evals.jsonl`). Must be a non-empty string; no duplicates allowed. +- `question` — the binary question the LLM judge will answer YES or NO. + +### Extended form — with model and runs-on overrides + +```yaml +evals: + questions: + - id: compiles + question: Does the generated code compile without errors? + - id: tests_pass + question: Do all existing tests still pass according to the agent output? + - id: scoped_change + question: Is the implementation limited to the scope described in the issue? + model: gpt-4o # per-question model override + model: small # default model for all questions + runs-on: ubuntu-latest +``` + +**Fields:** + +- `questions:` — list of question objects (required in extended form, ≥ 1 entry). +- `model:` — default LLM model for all questions. Use a model alias (`small`, `gpt-4o`) or a full model ID. Defaults to the engine's detection model (typically a small, cost-effective model). +- `runs-on:` — optional runner override for the evals job. Inherits the workflow default when omitted. + +Each question object may include its own `model:` field to override the top-level default for that question only. + +--- + +## Decomposing a Task into Binary Questions + +BinEval questions must be answerable with a strict YES or NO by an LLM reading the agent's output alone. Follow this process: + +### 1 — State the goal + +Write one sentence describing what a successful run looks like. + +> "The agent should update the CHANGELOG and bump the version number without touching unrelated files." + +### 2 — Identify observable properties + +Break the goal into properties that a judge can verify from `agent_output.json` and `prompt.txt`: + +| Property | Observable signal | +|---|---| +| CHANGELOG updated | Agent output mentions or contains CHANGELOG edits | +| Version bumped | A version number appears changed in the diff or agent summary | +| No unrelated files changed | Agent output does not list changes outside CHANGELOG and version files | + +### 3 — Write falsifiable YES/NO questions + +Each question should: + +- Be answerable YES when the property holds, NO otherwise. +- Reference observable evidence in the agent output — not intent or effort. +- Cover exactly one property (no compound questions with "and" or "or"). + +```yaml +evals: + - id: changelog_updated + question: Does the agent output confirm that CHANGELOG was updated? + - id: version_bumped + question: Does the agent output confirm that the version number was incremented? + - id: no_unrelated_files + question: Does the agent output show that only CHANGELOG and version files were modified? +``` + +### 4 — Assign question cost + +Prefer `model: small` (the default) for factual YES/NO checks. Reserve a larger model for questions that require nuanced reasoning: + +```yaml +evals: + questions: + - id: changelog_updated + question: Does the agent output confirm that CHANGELOG was updated? + - id: design_sound + question: Is the agent's proposed design consistent with established patterns in the codebase? + model: gpt-4o # nuanced, benefits from a larger model + model: small +``` + +### Good question checklist + +- ✅ Answerable from the agent output alone — no external calls needed. +- ✅ Exactly one binary claim per question. +- ✅ Uses YES = success convention consistently. +- ✅ Avoids subjective terms ("good", "well-written") unless the question explicitly bounds them ("according to the coding style guide"). +- ✅ Each question has a unique `id`. + +--- + +## Result Storage + +### Artifact + +Each run uploads `evals.jsonl` as the `evals` artifact (30-day retention). Each line is a JSON object: + +```json +{"run_id":"12345678","workflow_id":"my-workflow","id":"compiles","question":"Does the generated code compile?","answer":"YES","timestamp":"2026-07-15T10:00:00Z"} +``` + +### Git branch + +Results are also committed to `evals/` by the `push_evals_state` job (requires `contents: write`). This enables historical comparison across runs even after artifact expiry. + +Read results with: + +```bash +gh aw audit # includes evals section when present +gh aw logs # aggregates evals.jsonl across recent runs +``` + +--- + +## Required Permissions + +The evals job itself reads artifacts and runs the engine — no extra permissions beyond `contents: read`. The `push_evals_state` job that persists results to a git branch needs: + +```yaml +permissions: + contents: write +``` + +This is added automatically when `evals:` is declared. + +--- + +## Minimal Working Example + +```markdown +--- +description: Triage new issues and apply labels +on: + issues: + types: [opened] +engine: copilot +permissions: + issues: write +tools: + github: + toolsets: [issues] +safe-outputs: + add-label: + allowed-labels: [bug, enhancement, question, needs-triage] +evals: + - id: label_applied + question: Did the agent apply at least one label to the issue? + - id: correct_type + question: Is the applied label appropriate for the issue content described in the prompt? + - id: no_extra_labels + question: Did the agent avoid applying more than two labels? +--- + +Read ${{ github.event.issue.title }} and ${{ github.event.issue.body }}. +Apply the most appropriate label(s) from the allowed set. +``` + +Compile and deploy: + +```bash +gh aw compile issue-triage +``` + +--- + +## Anti-Patterns + +- ❌ **Compound questions** — "Did the agent update CHANGELOG and bump the version?" splits into two questions. A single NO is ambiguous. +- ❌ **Unobservable questions** — "Did the agent try its best?" cannot be answered from output text. +- ❌ **Duplicate IDs** — `id` must be unique within a workflow; the compiler rejects duplicates. +- ❌ **Empty questions** — both `id` and `question` must be non-empty strings. +- ❌ **Using a frontier model for all questions** — factual checks are cheap on small models; save larger models for reasoning-heavy questions. +- ❌ **Removing `evals:` mid-experiment** — breaks historical trend comparisons stored in the `evals/` branch. +- ❌ **Questions that require tool calls** — the evals engine runs in a network-restricted sandbox with only `bash`. Questions must be answerable from the downloaded agent artifact. diff --git a/.github/aw/syntax.md b/.github/aw/syntax.md index 765d0f76ce2..0a87d00948c 100644 --- a/.github/aw/syntax.md +++ b/.github/aw/syntax.md @@ -13,6 +13,7 @@ Use the smallest relevant reference instead of loading one large schema file. | Cache configuration, tools, imports, and permission patterns | [syntax-tools-imports.md](syntax-tools-imports.md) | | `skills` field | [skills.md](skills.md) | | `lsp` field | [lsp.md](lsp.md) | +| `evals` field (BinEval binary evaluations) | [evals.md](evals.md) | ## Usage Guidance diff --git a/.github/skills/agentic-workflows/SKILL.md b/.github/skills/agentic-workflows/SKILL.md index 499e6113b76..4c2b5396a97 100644 --- a/.github/skills/agentic-workflows/SKILL.md +++ b/.github/skills/agentic-workflows/SKILL.md @@ -31,6 +31,7 @@ Load these files from `github/gh-aw` (they are not available locally). - `.github/aw/dependabot.md` - `.github/aw/deployment-status.md` - `.github/aw/designer.md` +- `.github/aw/evals.md` - `.github/aw/experiments.md` - `.github/aw/github-agentic-workflows.md` - `.github/aw/github-mcp-server.md` diff --git a/pkg/cli/data/agentic_workflows_fallback_aw_files.json b/pkg/cli/data/agentic_workflows_fallback_aw_files.json index 8685f237053..e0f3f1cd08f 100644 --- a/pkg/cli/data/agentic_workflows_fallback_aw_files.json +++ b/pkg/cli/data/agentic_workflows_fallback_aw_files.json @@ -15,6 +15,7 @@ "dependabot.md", "deployment-status.md", "designer.md", + "evals.md", "experiments.md", "github-agentic-workflows.md", "github-mcp-server.md", From d342759b50cd410bcee4f22c56bf160ecc8ace19 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Thu, 16 Jul 2026 02:02:58 +0000 Subject: [PATCH 2/4] plan: fix evals.md documentation issues identified in review Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com> --- .github/skills/agentic-workflows/SKILL.md | 1 - 1 file changed, 1 deletion(-) diff --git a/.github/skills/agentic-workflows/SKILL.md b/.github/skills/agentic-workflows/SKILL.md index 4c2b5396a97..499e6113b76 100644 --- a/.github/skills/agentic-workflows/SKILL.md +++ b/.github/skills/agentic-workflows/SKILL.md @@ -31,7 +31,6 @@ Load these files from `github/gh-aw` (they are not available locally). - `.github/aw/dependabot.md` - `.github/aw/deployment-status.md` - `.github/aw/designer.md` -- `.github/aw/evals.md` - `.github/aw/experiments.md` - `.github/aw/github-agentic-workflows.md` - `.github/aw/github-mcp-server.md` From 3f7f87e5ab917f6f83137bf49705c11425b20261 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Thu, 16 Jul 2026 02:12:17 +0000 Subject: [PATCH 3/4] =?UTF-8?q?docs:=20fix=20evals.md=20review=20issues=20?= =?UTF-8?q?=E2=80=94=20accurate=20implementation=20alignment?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Remove prompt.txt from Setup step: run_evals.cjs only reads agent_output.json - Remove "optionally" from Upload step: push_evals_state always runs when evals declared - Add safe-outputs prerequisite note to shorthand example; agent_output.json is only included in agent artifact when safe-outputs is declared (compiler_yaml_post_agent.go:65-69) - Fix shorthand example questions to reference agent output, not issue context - Fix runs-on default to ubuntu-latest (not workflow default) - Remove per-question model override docs (marshalEvalsQuestions drops Model field) - Fix observable properties preamble to drop prompt.txt reference - Fix assign-question-cost example to use top-level model override only - Fix artifact retention: follows repo/org settings, not fixed 30 days - Fix JSON schema to match actual run_evals.cjs records (model, runid, not run_id/workflow_id) - Fix CLI examples to show actual --artifacts evals / --evals flags - Fix permissions to mention conditional copilot-requests:write/id-token:write - Fix minimal example label questions to say "requested via safe-output action" (evals runs parallel with safe_outputs, cannot verify applied labels) Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com> --- .github/aw/evals.md | 62 +++++++++++++---------- .github/skills/agentic-workflows/SKILL.md | 1 + 2 files changed, 35 insertions(+), 28 deletions(-) diff --git a/.github/aw/evals.md b/.github/aw/evals.md index 1a3ddf84aaf..c760af1e0ce 100644 --- a/.github/aw/evals.md +++ b/.github/aw/evals.md @@ -12,11 +12,11 @@ Evals let you verify automatically whether an agentic run met its goals. Each ev Per run: -1. **Setup** — the evals job downloads the agent artifact (`agent_output.json`, `prompt.txt`) and writes a BinEval prompt containing all declared questions. +1. **Setup** — the evals job downloads the agent artifact (`agent_output.json`) and writes a BinEval prompt containing all declared questions. 2. **Execute** — an LLM judge runs in a network-restricted sandbox (same engine as the agent job) and answers each question with YES or NO. 3. **Parse** — raw engine output is parsed into per-question records and written to `evals.jsonl`. 4. **Redact** — any credential patterns are removed from the results before upload. -5. **Upload** — `evals.jsonl` is uploaded as the `evals` workflow artifact and optionally persisted to the `evals/` git branch. +5. **Upload** — `evals.jsonl` is uploaded as the `evals` workflow artifact and committed to the `evals/` git branch by the `push_evals_state` job. The evals job runs **after** the agent job and **in parallel with** `safe_outputs`, so it does not block the write path. @@ -26,17 +26,22 @@ The evals job runs **after** the agent job and **in parallel with** `safe_output ### Shorthand — plain list +> **Prerequisite:** `agent_output.json` is only included in the agent artifact when `safe-outputs` is also declared. Without it the evals job runs with no agent context and every question will receive `UNKNOWN`. + ```yaml --- on: issues: types: [opened] engine: copilot +safe-outputs: + comment: + allowed-tools: ["*"] evals: - - id: scoped_change - question: Is the implementation limited to the scope described in the issue? - - id: no_regressions - question: Does the change avoid modifying files unrelated to the task? + - id: response_provided + question: Does the agent output confirm that a response was written? + - id: no_unrelated_files + question: Does the agent output show that only the expected files were modified? --- Implement the requested change described in ${{ github.event.issue.body }}. @@ -57,19 +62,16 @@ evals: - id: tests_pass question: Do all existing tests still pass according to the agent output? - id: scoped_change - question: Is the implementation limited to the scope described in the issue? - model: gpt-4o # per-question model override - model: small # default model for all questions + question: Does the agent output show that only the expected files were modified? + model: small # model for all questions runs-on: ubuntu-latest ``` **Fields:** - `questions:` — list of question objects (required in extended form, ≥ 1 entry). -- `model:` — default LLM model for all questions. Use a model alias (`small`, `gpt-4o`) or a full model ID. Defaults to the engine's detection model (typically a small, cost-effective model). -- `runs-on:` — optional runner override for the evals job. Inherits the workflow default when omitted. - -Each question object may include its own `model:` field to override the top-level default for that question only. +- `model:` — LLM model for all questions. Use a model alias (`small`, `gpt-4o`) or a full model ID. Defaults to the engine's detection model (typically a small, cost-effective model). +- `runs-on:` — optional runner override for the evals job. Defaults to `ubuntu-latest` when omitted. --- @@ -85,7 +87,7 @@ Write one sentence describing what a successful run looks like. ### 2 — Identify observable properties -Break the goal into properties that a judge can verify from `agent_output.json` and `prompt.txt`: +Break the goal into properties that a judge can verify from `agent_output.json`: | Property | Observable signal | |---|---| @@ -113,7 +115,7 @@ evals: ### 4 — Assign question cost -Prefer `model: small` (the default) for factual YES/NO checks. Reserve a larger model for questions that require nuanced reasoning: +Prefer `model: small` (the default) for factual YES/NO checks. Reserve a larger model for questions that require nuanced reasoning by setting `model` at the `evals:` level: ```yaml evals: @@ -121,9 +123,8 @@ evals: - id: changelog_updated question: Does the agent output confirm that CHANGELOG was updated? - id: design_sound - question: Is the agent's proposed design consistent with established patterns in the codebase? - model: gpt-4o # nuanced, benefits from a larger model - model: small + question: Is the agent's proposed design consistent with established patterns described in the agent output? + model: gpt-4o # nuanced questions; override default small model ``` ### Good question checklist @@ -140,10 +141,10 @@ evals: ### Artifact -Each run uploads `evals.jsonl` as the `evals` artifact (30-day retention). Each line is a JSON object: +Each run uploads `evals.jsonl` as the `evals` artifact (retention follows repository or organization settings). Each line is a JSON object: ```json -{"run_id":"12345678","workflow_id":"my-workflow","id":"compiles","question":"Does the generated code compile?","answer":"YES","timestamp":"2026-07-15T10:00:00Z"} +{"id":"compiles","question":"Does the generated code compile?","answer":"YES","model":"small","timestamp":"2026-07-15T10:00:00Z","runid":"12345678"} ``` ### Git branch @@ -153,15 +154,20 @@ Results are also committed to `evals/` by the `push_evals Read results with: ```bash -gh aw audit # includes evals section when present -gh aw logs # aggregates evals.jsonl across recent runs +gh aw audit --artifacts evals # downloads evals.jsonl from the run artifact +gh aw logs --evals # filter to runs that contain evals results ``` --- ## Required Permissions -The evals job itself reads artifacts and runs the engine — no extra permissions beyond `contents: read`. The `push_evals_state` job that persists results to a git branch needs: +The evals job itself reads artifacts and runs the engine. The compiler grants `contents: read` by default, and conditionally adds: + +- `copilot-requests: write` — when the workflow uses Copilot API requests. +- `id-token: write` — when the workflow uses GitHub OIDC authentication or OTLP telemetry. + +These are added automatically; no manual configuration is needed. The `push_evals_state` job that persists results to a git branch always needs: ```yaml permissions: @@ -190,12 +196,12 @@ safe-outputs: add-label: allowed-labels: [bug, enhancement, question, needs-triage] evals: - - id: label_applied - question: Did the agent apply at least one label to the issue? - - id: correct_type - question: Is the applied label appropriate for the issue content described in the prompt? + - id: label_requested + question: Does the agent output show that at least one label was requested via a safe-output action? + - id: label_in_allowed_set + question: Does the agent output show that the requested label belongs to the allowed set (bug, enhancement, question, needs-triage)? - id: no_extra_labels - question: Did the agent avoid applying more than two labels? + question: Does the agent output show that no more than two labels were requested? --- Read ${{ github.event.issue.title }} and ${{ github.event.issue.body }}. diff --git a/.github/skills/agentic-workflows/SKILL.md b/.github/skills/agentic-workflows/SKILL.md index 499e6113b76..4c2b5396a97 100644 --- a/.github/skills/agentic-workflows/SKILL.md +++ b/.github/skills/agentic-workflows/SKILL.md @@ -31,6 +31,7 @@ Load these files from `github/gh-aw` (they are not available locally). - `.github/aw/dependabot.md` - `.github/aw/deployment-status.md` - `.github/aw/designer.md` +- `.github/aw/evals.md` - `.github/aw/experiments.md` - `.github/aw/github-agentic-workflows.md` - `.github/aw/github-mcp-server.md` From 8180c04f9e4493ee10b544100a0942baca7387eb Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Thu, 16 Jul 2026 02:13:12 +0000 Subject: [PATCH 4/4] docs: fix punctuation and clarify conditional permissions note in evals.md Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com> --- .github/aw/evals.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/aw/evals.md b/.github/aw/evals.md index c760af1e0ce..9f99869bcaf 100644 --- a/.github/aw/evals.md +++ b/.github/aw/evals.md @@ -26,7 +26,7 @@ The evals job runs **after** the agent job and **in parallel with** `safe_output ### Shorthand — plain list -> **Prerequisite:** `agent_output.json` is only included in the agent artifact when `safe-outputs` is also declared. Without it the evals job runs with no agent context and every question will receive `UNKNOWN`. +> **Prerequisite:** `agent_output.json` is only included in the agent artifact when `safe-outputs` is also declared. Without it, the evals job runs with no agent context and every question will receive `UNKNOWN`. ```yaml --- @@ -162,7 +162,7 @@ gh aw logs --evals # filter to runs that contain evals re ## Required Permissions -The evals job itself reads artifacts and runs the engine. The compiler grants `contents: read` by default, and conditionally adds: +The evals job itself reads artifacts and runs the engine. The compiler grants `contents: read` by default, and conditionally adds the following when the corresponding features are used in the workflow: - `copilot-requests: write` — when the workflow uses Copilot API requests. - `id-token: write` — when the workflow uses GitHub OIDC authentication or OTLP telemetry.