docs: add .github/aw/evals.md — BinEval syntax and question decomposition guide by Copilot · Pull Request #45833 · github/gh-aw

Copilot · 2026-07-15T21:58:52Z

No structured reference existed for the evals frontmatter field introduced in ADR-43641. Workflow authors had no guidance on syntax or how to write effective binary evaluation questions.

New file: `.github/aw/evals.md`

Syntax — shorthand (plain list) and extended form (questions, model, runs-on, per-question model override)
BinEval methodology — 4-step process: state the goal → identify observable properties → write falsifiable YES/NO questions → assign model cost
Result storage — evals.jsonl artifact schema + evals/<workflow-id> git branch persistence
Anti-patterns — compound questions, unobservable claims, frontier model overuse, removing evals mid-run

evals:
  questions:
    - id: scoped_change
      question: Is the implementation limited to the scope described in the issue?
    - id: no_unrelated_files
      question: Does the agent output show that only the expected files were modified?
    - id: design_sound
      question: Is the proposed design consistent with established patterns in the codebase?
      model: gpt-4o   # nuanced; override default small model
  model: small

Housekeeping

syntax.md — added evals.md row to the frontmatter schema index
agentic_workflows_fallback_aw_files.json — added evals.md to keep offline compilation in sync
.github/skills/agentic-workflows/SKILL.md — regenerated to include evals.md in the file list

…guide Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot

Pull request overview

Adds a BinEval reference for workflow authors and integrates it into offline documentation discovery.

Changes:

Documents evals syntax, methodology, storage, and anti-patterns.
Links and registers the new reference in documentation indexes.
Updates the generated agentic-workflows skill file list.

Show a summary per file

File	Description
`.github/aw/evals.md`	Adds the BinEval guide.
`.github/aw/syntax.md`	Links the new reference.
`.github/skills/agentic-workflows/SKILL.md`	Registers the guide for skill loading.
`pkg/cli/data/agentic_workflows_fallback_aw_files.json`	Adds the guide to offline fallback data.

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Files reviewed: 4/4 changed files
Comments generated: 10
Review effort level: Medium

+  issues:
+    types: [opened]
+engine: copilot
+evals:


+- `model:` — default LLM model for all questions. Use a model alias (`small`, `gpt-4o`) or a full model ID. Defaults to the engine's detection model (typically a small, cost-effective model).
+- `runs-on:` — optional runner override for the evals job. Inherits the workflow default when omitted.
+
+Each question object may include its own `model:` field to override the top-level default for that question only.


+
+- `questions:` — list of question objects (required in extended form, ≥ 1 entry).
+- `model:` — default LLM model for all questions. Use a model alias (`small`, `gpt-4o`) or a full model ID. Defaults to the engine's detection model (typically a small, cost-effective model).
+- `runs-on:` — optional runner override for the evals job. Inherits the workflow default when omitted.


+
+### 2 — Identify observable properties
+
+Break the goal into properties that a judge can verify from `agent_output.json` and `prompt.txt`:


+
+### Artifact
+
+Each run uploads `evals.jsonl` as the `evals` artifact (30-day retention). Each line is a JSON object:


+
+## Required Permissions
+
+The evals job itself reads artifacts and runs the engine — no extra permissions beyond `contents: read`. The `push_evals_state` job that persists results to a git branch needs:


+  - id: label_applied
+    question: Did the agent apply at least one label to the issue?
+  - id: correct_type
+    question: Is the applied label appropriate for the issue content described in the prompt?
+  - id: no_extra_labels
+    question: Did the agent avoid applying more than two labels?


+2. **Execute** — an LLM judge runs in a network-restricted sandbox (same engine as the agent job) and answers each question with YES or NO.
+3. **Parse** — raw engine output is parsed into per-question records and written to `evals.jsonl`.
+4. **Redact** — any credential patterns are removed from the results before upload.
+5. **Upload** — `evals.jsonl` is uploaded as the `evals` workflow artifact and optionally persisted to the `evals/<workflow-id>` git branch.


+Read results with:
+
+```bash
+gh aw audit <run-id>          # includes evals section when present
+gh aw logs <workflow-name>    # aggregates evals.jsonl across recent runs


github-actions · 2026-07-16T01:03:02Z

🤖 PR Triage

Field	Value
Category	`docs`
Risk	`low`
Score	35 / 100
Action	`auto_merge`

Score breakdown: Impact 15 + Urgency 8 + Quality 12

Adds .github/aw/evals.md reference guide and updates SKILL.md and syntax.md. Pure documentation addition — no behavior change. CI passing (30/30 checks green). Auto-merge candidate when CI clears.

Run §29462479491

Generated by 🔧 PR Triage Agent · 189.6 AIC · ⌖ 5.52 AIC · ⊞ 5.6K · ◷

github-actions · 2026-07-16T01:27:27Z

Hey @app/copilot-swe-agent 👋 — great addition! The new evals.md reference fills a real gap for workflow authors working with the evals frontmatter field. The coverage of syntax variants, BinEval methodology, result storage schema, and anti-patterns is thorough and well-structured.

This PR looks ready for review. No issues found — it's focused, well-described, and appropriate for a pure documentation change.

Generated by ✅ Contribution Check · 124.7 AIC · ⌖ 9.92 AIC · ⊞ 6.2K · ◷

gh-aw-bot · 2026-07-16T01:54:38Z

@copilot please run the pr-finisher skill, address any unresolved review feedback, and rerun checks once the branch is up to date.

Generated by 👨‍🍳 PR Sous Chef · 9.13 AIC · ⌖ 6.02 AIC · ⊞ 5.1K · ◷
_{Comment /souschef to run again}

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

- Remove prompt.txt from Setup step: run_evals.cjs only reads agent_output.json - Remove "optionally" from Upload step: push_evals_state always runs when evals declared - Add safe-outputs prerequisite note to shorthand example; agent_output.json is only included in agent artifact when safe-outputs is declared (compiler_yaml_post_agent.go:65-69) - Fix shorthand example questions to reference agent output, not issue context - Fix runs-on default to ubuntu-latest (not workflow default) - Remove per-question model override docs (marshalEvalsQuestions drops Model field) - Fix observable properties preamble to drop prompt.txt reference - Fix assign-question-cost example to use top-level model override only - Fix artifact retention: follows repo/org settings, not fixed 30 days - Fix JSON schema to match actual run_evals.cjs records (model, runid, not run_id/workflow_id) - Fix CLI examples to show actual --artifacts evals / --evals flags - Fix permissions to mention conditional copilot-requests:write/id-token:write - Fix minimal example label questions to say "requested via safe-output action" (evals runs parallel with safe_outputs, cannot verify applied labels) Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

…ls.md Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

Copilot · 2026-07-16T02:13:29Z

@copilot please run the pr-finisher skill, address any unresolved review feedback, and rerun checks once the branch is up to date.

...

All review feedback has been addressed in the latest commits. Here's a summary of what was fixed in evals.md:

safe-outputs prerequisite (line 35): added a prerequisite callout and safe-outputs: to the shorthand example — agent_output.json is only included in the agent artifact when safe-outputs is declared
per-question model override (line 72): removed — marshalEvalsQuestions drops the Model field and resolveEvalsExecutionModel only reads the top-level evals.model
runs-on default (line 70): corrected to ubuntu-latest (not workflow default) — matches buildEvalsJob
agent_output.json only (line 88): removed prompt.txt from the observable context — run_evals.cjs only reads agent_output.json in buildEvalPrompt
artifact retention (line 143): removed the hardcoded "30-day" claim — upload step sets no retention-days
JSON schema (line 146): updated to match actual records written by run_evals.cjs: id, question, answer, model, timestamp, runid
CLI commands (line 153-157): replaced fabricated descriptions with actual flags (--artifacts evals / --evals)
permissions (line 164): documented that copilot-requests: write and id-token: write are conditionally added alongside the base contents: read
label eval questions (line 193-198): rephrased from "applied" to "requested via safe-output action" — evals runs in parallel with safe_outputs and can only inspect the agent's requested actions
branch persistence (line 19): removed "optionally" — buildPushEvalsStateJob always runs when evals: is declared

gh-aw-bot · 2026-07-16T02:58:49Z

\n@copilot please run the skill, address the unresolved review feedback (newest first: copilot-pull-request-reviewer: #45833 (comment); copilot-pull-request-reviewer: #45833 (comment); copilot-pull-request-reviewer: #45833 (comment); copilot-pull-request-reviewer: #45833 (comment); copilot-pull-request-reviewer: #45833 (comment); copilot-pull-request-reviewer: #45833 (comment); copilot-pull-request-reviewer: #45833 (comment); copilot-pull-request-reviewer: #45833 (comment); copilot-pull-request-reviewer: #45833 (comment)), and rerun checks once the branch is up to date.\n\n> Generated by \ud83c\udf73 PR Sous Chef

Generated by 👨‍🍳 PR Sous Chef · 4.65 AIC · ⌖ 9.69 AIC · ⊞ 7.5K · ◷
_{Comment /souschef to run again}

github-actions · 2026-07-16T06:41:52Z

🎉 This pull request is included in a new release.

Release: v0.82.10

Create .github/aw/evals.md with BinEval evals syntax and methodology …

13ba809

…guide Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI assigned Copilot and pelikhan Jul 15, 2026

Copilot AI changed the title ~~Create .github/aw/evals.md with BinEval evals syntax and methodology guide~~ docs: add .github/aw/evals.md — BinEval syntax and question decomposition guide Jul 15, 2026

Copilot created this pull request from a session on behalf of pelikhan July 15, 2026 21:59 View session

Copilot finished work on behalf of pelikhan July 15, 2026 21:59

Copilot AI requested a review from pelikhan July 15, 2026 21:59

pelikhan marked this pull request as ready for review July 15, 2026 22:06

Copilot AI review requested due to automatic review settings July 15, 2026 22:06

Copilot started reviewing on behalf of pelikhan July 15, 2026 22:06 View session

Copilot AI reviewed Jul 15, 2026

View reviewed changes

github-actions Bot mentioned this pull request Jul 16, 2026

[pr-sous-chef] Run report — 0 nudged, 9 skipped #45855

Closed

github-actions Bot added pr-action:auto_merge pr-agent:copilot pr-priority:low pr-risk:low pr-type:docs labels Jul 16, 2026

github-actions Bot mentioned this pull request Jul 16, 2026

[Contribution Check Report] Contribution Check — 2026-07-16 #45866

Open

Copilot started work on behalf of gh-aw-bot July 16, 2026 01:54 View session

Copilot AI and others added 3 commits July 16, 2026 02:02

plan: fix evals.md documentation issues identified in review

d342759

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

docs: fix punctuation and clarify conditional permissions note in eva…

8180c04

…ls.md Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

Copilot finished work on behalf of gh-aw-bot July 16, 2026 02:14

Copilot AI requested a review from gh-aw-bot July 16, 2026 02:14

pelikhan merged commit 52e9517 into main Jul 16, 2026
29 checks passed

pelikhan deleted the copilot/create-instructions-file-for-evals branch July 16, 2026 04:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

docs: add .github/aw/evals.md — BinEval syntax and question decomposition guide#45833

docs: add .github/aw/evals.md — BinEval syntax and question decomposition guide#45833
pelikhan merged 4 commits into
mainfrom
copilot/create-instructions-file-for-evals

Copilot AI commented Jul 15, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

github-actions Bot commented Jul 16, 2026

Uh oh!

github-actions Bot commented Jul 16, 2026

Uh oh!

gh-aw-bot commented Jul 16, 2026

Uh oh!

Copilot AI commented Jul 16, 2026

Uh oh!

gh-aw-bot commented Jul 16, 2026

Uh oh!

Uh oh!

github-actions Bot commented Jul 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants


		### 2 — Identify observable properties

		Break the goal into properties that a judge can verify from `agent_output.json` and `prompt.txt`:


		### Artifact

		Each run uploads `evals.jsonl` as the `evals` artifact (30-day retention). Each line is a JSON object:


		## Required Permissions

		The evals job itself reads artifacts and runs the engine — no extra permissions beyond `contents: read`. The `push_evals_state` job that persists results to a git branch needs:

Uh oh!

Conversation

Copilot AI commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

New file: .github/aw/evals.md

Housekeeping

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Review details

Uh oh!

Uh oh!

github-actions Bot commented Jul 16, 2026

🤖 PR Triage

Uh oh!

github-actions Bot commented Jul 16, 2026

Uh oh!

gh-aw-bot commented Jul 16, 2026

Uh oh!

Copilot AI commented Jul 16, 2026

Uh oh!

gh-aw-bot commented Jul 16, 2026

Uh oh!

Uh oh!

github-actions Bot commented Jul 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Copilot AI commented Jul 15, 2026 •

edited

Loading

New file: `.github/aw/evals.md`