Enable evals on 20 more agentic workflows by Copilot · Pull Request #49078 · github/gh-aw

Copilot · 2026-07-30T06:36:08Z

Expands automated outcome evaluation to 20 additional agentic workflows.

Evaluation coverage
- Adds two workflow-specific binary checks per workflow.
- Covers task execution and expected output or noop behavior.
Generated workflows
- Recompiles affected lock files with eval execution and result persistence jobs.

evals:
  - id: action_versions_checked
    question: Did the agent check GitHub Actions versions for available updates?
  - id: update_pr_created_or_noop
    question: Did the agent create a pull request for required updates, or report that no changes were needed?

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

pelikhan · 2026-07-30T06:37:02Z

@copilot resolve the merge conflicts on this branch.

Copilot

Pull request overview

Adds BinEval outcome evaluation to 20 agentic workflows and updates compiled workflows to execute, store, and report evaluation results.

Changes:

Adds two workflow-specific evaluation questions per workflow.
Adds evaluation jobs, artifacts, AI-credit reporting, and persisted result branches.
Covers task execution and terminal outcomes such as reports, pull requests, issues, and noops.

Show a summary per file

File	Description
`.github/workflows/functional-pragmatist.md`	Adds functional-refactoring evals.
`.github/workflows/functional-pragmatist.lock.yml`	Compiles evaluation and persistence jobs.
`.github/workflows/eslint-refiner.md`	Adds ESLint refinement evals.
`.github/workflows/eslint-monster.md`	Adds diagnostics/remediation evals.
`.github/workflows/eslint-monster.lock.yml`	Compiles ESLint Monster eval jobs.
`.github/workflows/eslint-miner.md`	Adds rule-mining evals.
`.github/workflows/dictation-prompt.md`	Adds dictation-generation evals.
`.github/workflows/developer-docs-consolidator.md`	Adds documentation-consolidation evals.
`.github/workflows/dev.md`	Adds daily status evals.
`.github/workflows/detection-analysis-report.md`	Adds detection-analysis evals.
`.github/workflows/dependabot-burner.md`	Adds remediation-batch evals.
`.github/workflows/daily-workflow-updater.md`	Adds action-update evals.
`.github/workflows/daily-windows-terminal-integration-builder.md`	Adds Windows integration evals.
`.github/workflows/daily-windows-terminal-integration-builder.lock.yml`	Compiles Windows integration eval jobs.
`.github/workflows/daily-token-consumption-report.md`	Adds consumption-report evals.
`.github/workflows/daily-squid-image-scan.md`	Adds container-scan evals.
`.github/workflows/daily-squid-image-scan.lock.yml`	Compiles image-scan eval jobs.
`.github/workflows/daily-spending-forecast.md`	Adds spending-forecast evals.
`.github/workflows/daily-spending-forecast.lock.yml`	Compiles forecast eval jobs.
`.github/workflows/daily-spdd-spec-planner.md`	Adds specification-planning evals.
`.github/workflows/daily-safeoutputs-git-simulator.md`	Adds git-simulator evals.
`.github/workflows/daily-multi-device-docs-tester.md`	Adds multi-device testing evals.
`.github/workflows/daily-multi-device-docs-tester.lock.yml`	Compiles device-testing eval jobs.
`.github/workflows/daily-go-test-parallelizer.md`	Adds test-parallelization evals.
`.github/workflows/daily-go-test-parallelizer.lock.yml`	Compiles parallelizer eval jobs.
`.github/workflows/daily-github-docs-seo-optimizer.md`	Adds documentation SEO evals.
`.github/workflows/daily-awf-spec-compiler-surfacing.md`	Adds AWF surfacing evals.
`.github/workflows/daily-awf-spec-compiler-surfacing.lock.yml`	Compiles AWF surfacing eval jobs.

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comments suppressed due to low confidence (4)

.github/workflows/dictation-prompt.md:55

This check rejects the workflow's valid no-change path: the prompt explicitly requires noop when DICTATION.md needs no meaningful update, but this evaluation can pass only when a pull request is created. Include the noop outcome so successful no-change runs are not scored as failures, then recompile the lock file.
.github/workflows/detection-analysis-report.md:42
The workflow explicitly calls noop when there are no runs in the 24-hour window, yet this criterion requires a report. Such a valid empty-window run will be scored as a failed output check; accept the documented noop outcome and recompile the lock file.

  - id: detection_report_created
    question: Did the agent create a report with evidence-backed detection findings or recommendations?

.github/workflows/dictation-prompt.md:55

The evals job runs concurrently with safe_outputs and can inspect only the agent artifact, so actual pull-request creation is not observable (.github/aw/evals.md:13-21). Rephrase the criterion to verify that the output requested a dictation-update PR; this is separate from also accepting the documented noop path.
.github/workflows/daily-multi-device-docs-tester.md:137
This can return YES for a prose-only report even though the prompt states that the workflow fails unless create-issue or noop is actually called. Evaluate whether the required terminal safe-output request appears in agent_output.json, rather than report text alone, and regenerate the lock file.

  - id: results_reported
    question: Did the agent report the multi-device test results and any responsive design or functionality findings?

Files reviewed: 40/40 changed files
Comments generated: 22
Review effort level: Medium

+  - id: status_report_created
+    question: Did the agent create a daily status report with relevant project updates?


+  - id: consumption_report_created
+    question: Did the agent create a report with token consumption trends and actionable observations?


+  - id: refactoring_pr_created_or_noop
+    question: Did the agent create a focused refactoring pull request, or report that no suitable change was found?


+  - id: remediation_dispatched_or_noop
+    question: Did the agent dispatch remediation for actionable findings, or use noop when the scan was clean?


+  - id: rule_pr_created_or_noop
+    question: Did the agent create a pull request for a new ESLint rule, or use noop when no suitable rule was found?


+  - id: dictation_prompt_generated
+    question: Did the agent generate an optimized prompt for voice dictation and speech-to-text workflows?


+  - id: refinements_reported
+    question: Did the agent report actionable ESLint rule refinements or explain why no refinement was needed?


+  - id: prioritized_plan_reported
+    question: Did the agent report a prioritized plan with actionable specification work items?


+  - id: recommendation_reported
+    question: Did the agent report an actionable documentation recommendation or explain why no update was needed?


+  - id: project_activity_analyzed
+    question: Did the agent analyze recent gh-aw project activity for the daily status report?


…on-agentic-workflows # Conflicts: # .github/workflows/dependabot-burner.lock.yml Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

gh-aw-bot · 2026-07-30T06:51:23Z

@copilot this branch has merge conflicts. Please run make merge-main to merge the latest main branch and resolve any conflicts, then push the result. After that, run the pr-finisher skill and summarize any remaining reviewer or CI follow-up.

Run: https://github.com/github/gh-aw/actions/runs/30520508562

Generated by 👨‍🍳 PR Sous Chef · gpt54 · 7.12 AIC · ⌖ 6.35 AIC · ⊞ 8.7K · ◷
_{Comment /souschef to run again}

Copilot · 2026-07-30T07:00:30Z

@copilot resolve the merge conflicts on this branch.

Resolved the merge conflict with current main in merge commit 814ffc9. The generated workflow was recompiled from the merged source, and final validation passes.

github-actions · 2026-07-31T14:16:41Z

🎉 This pull request is included in a new release.

Release: v0.84.1

Enable evals on 20 more workflows

e7a1629

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI assigned Copilot and pelikhan Jul 30, 2026

Copilot finished work on behalf of pelikhan July 30, 2026 06:36

Copilot AI requested a review from pelikhan July 30, 2026 06:36

Copilot created this pull request from a session on behalf of pelikhan July 30, 2026 06:36 View session

pelikhan marked this pull request as ready for review July 30, 2026 06:36

Copilot AI review requested due to automatic review settings July 30, 2026 06:36

Copilot started reviewing on behalf of pelikhan July 30, 2026 06:37 View session

Copilot started work on behalf of pelikhan July 30, 2026 06:42 View session

Copilot AI reviewed Jul 30, 2026

View reviewed changes

Merge remote-tracking branch 'origin/main' into copilot/enable-evals-…

814ffc9

…on-agentic-workflows # Conflicts: # .github/workflows/dependabot-burner.lock.yml Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

github-actions Bot mentioned this pull request Jul 30, 2026

[pr-sous-chef] PR Sous Chef run report #49011

Closed

Copilot started work on behalf of gh-aw-bot July 30, 2026 06:52 View session

Copilot finished work on behalf of pelikhan July 30, 2026 07:00

pelikhan merged commit 8fcdd9f into main Jul 30, 2026

pelikhan deleted the copilot/enable-evals-on-agentic-workflows branch July 30, 2026 07:02

Copilot stopped work on behalf of gh-aw-bot due to an error July 30, 2026 07:02
The session was cancelled by the user.

Copilot AI requested a review from gh-aw-bot July 30, 2026 07:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Enable evals on 20 more agentic workflows - #49078

Enable evals on 20 more agentic workflows#49078
pelikhan merged 2 commits into
mainfrom
copilot/enable-evals-on-agentic-workflows

Copilot AI commented Jul 30, 2026 •

edited

Loading

Uh oh!

pelikhan commented Jul 30, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

gh-aw-bot commented Jul 30, 2026

Uh oh!

Copilot AI commented Jul 30, 2026

Uh oh!

github-actions Bot commented Jul 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		- id: status_report_created
		question: Did the agent create a daily status report with relevant project updates?

		- id: consumption_report_created
		question: Did the agent create a report with token consumption trends and actionable observations?

		- id: refactoring_pr_created_or_noop
		question: Did the agent create a focused refactoring pull request, or report that no suitable change was found?

		- id: remediation_dispatched_or_noop
		question: Did the agent dispatch remediation for actionable findings, or use noop when the scan was clean?

		- id: rule_pr_created_or_noop
		question: Did the agent create a pull request for a new ESLint rule, or use noop when no suitable rule was found?

		- id: dictation_prompt_generated
		question: Did the agent generate an optimized prompt for voice dictation and speech-to-text workflows?

		- id: refinements_reported
		question: Did the agent report actionable ESLint rule refinements or explain why no refinement was needed?

		- id: prioritized_plan_reported
		question: Did the agent report a prioritized plan with actionable specification work items?

		- id: recommendation_reported
		question: Did the agent report an actionable documentation recommendation or explain why no update was needed?

		- id: project_activity_analyzed
		question: Did the agent analyze recent gh-aw project activity for the daily status report?

Uh oh!

Conversation

Copilot AI commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pelikhan commented Jul 30, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Review details

Uh oh!

gh-aw-bot commented Jul 30, 2026

Uh oh!

Copilot AI commented Jul 30, 2026

Uh oh!

github-actions Bot commented Jul 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Copilot AI commented Jul 30, 2026 •

edited

Loading