[prompt-clustering] Copilot Agent Prompt Clustering Analysis – 959 PRs, 8 Clusters

[github-actions[bot]]

Summary

Analysis period: All available history (~959 copilot PRs)
Prompts extracted: 959
Clusters identified: 8
Overall merge success rate: 69.0%

Cluster	Theme	Size	Success Rate
C4	Agentic Workflow Updates & Features	406	74%
C3	CI Failure Doctor – Issue-Based	176	53%
C2	Issue-Tracker–Driven Tasks	103	58%
C8	Safe Outputs Infrastructure	84	81%
C1	Unclear / Meta Prompts	67	73%
C5	Maintenance & Logging Improvements	60	63%
C6	Direct CI Job Failure Fixes	33	82%
C7	Report Formatting Standardization	30	87%

Key Findings

1. Dominant task type: Agentic Workflow Updates & Features accounts for 406 PRs (42% of all tasks) — direct-text prompts for agentic workflow .md changes and MCP server updates.
2. Highest success rate: Report Formatting Standardization at 87% (C7) — narrow, well-specified tasks with a precise target file and rule produce reliably good outcomes.
3. Lowest success rate: CI Failure Doctor – Issue-Based at 53% (C3) — broader issue-tracker tasks with less specific acceptance criteria result in more unmerged PRs.
4. Prompt specificity correlates with success: Clusters with a direct action (job URL, version number, formatting rule) merge at 80–87%; open-ended issue-driven clusters hover at 52–63%.
5. Structured prompt templates help routing but not outcomes: Issue-driven clusters (C2, C3) use consistent XML-like tags, yet their success rates are lower — suggesting the bottleneck is task clarity, not prompt routing.

Cluster-by-Cluster Breakdown

C1: Unclear / Meta Prompts

• Size: 67 PRs  |  Success rate: 73% ███████░░░

• Top keywords: coding agent, coding, copilot coding, agent tips, tips, agent

• Description: PRs where the original prompt wasn't extracted cleanly — body contained only the copilot suffix section with tips/agent metadata rather than a clear task description.

• Example PRs: #11065, #11066, #11097

  Example prompt (#11065): Thanks for asking me to work on this. I will get started on it and keep this PR's description up to date as I form a plan and make progress. (prompt body not captured in PR)

---

C2: Issue-Tracker–Driven Tasks

• Size: 103 PRs  |  Success rate: 58% █████░░░░░

• Top keywords: task miner, miner, discussion task, discussion, task, gh aw

• Description: Tasks structured around <issue_title> / <issue_description> tags, typically spawned from the task-miner or discussion-task workflows. Broad scope: Go toolchain, MCP tool setup, workflow fixes.

• Example PRs: #11059, #11067, #11074

  Example prompt (#11059): The Daily CLI Performance Agent reports a missing_tool because the Go toolchain is not installed when it tries to run benchmark-related commands. Add an actions/setup-go@v5 step…

---

C3: CI Failure Doctor – Issue-Based

• Size: 176 PRs  |  Success rate: 53% █████░░░░░

• Top keywords: gh aw, aw, gh, issue_title, workflow, issue

• Description: CI failures or regressions packaged as structured issues and routed to Copilot. Examples: regex changes breaking tests, ANSI escape sequences in YAML, TypeScript type errors after prior PRs.

• Example PRs: #11058, #11068, #11069

  Example prompt (#11058): CI workflow failed after merging PR Fix expiration detection for quoted footers and legacy format #11036 which updated expiration detection regex patterns in ephemerals.cjs. The changes required blockquote prefixes…

---

C4: Agentic Workflow Updates & Features

• Size: 406 PRs  |  Success rate: 74% ███████░░░

• Top keywords: agentic, update, reference, campaign, workflows, workflow

• Description: The dominant category — direct-text prompts requesting changes to agentic workflow .md files, campaigns, MCP servers, or supporting infrastructure. Covers upgrades, new flags, and feature additions.

• Example PRs: #11050, #11053, #11054

  Example prompt (#11050): Run the update command and ensure that the Sentry MCP is updated. It should be upgraded to version 0.27.0, as shown in the dependency pull-request description attached below.

---

C5: Maintenance & Logging Improvements

• Size: 60 PRs  |  Success rate: 63% ██████░░░░

• Top keywords: comments, failure, workflow failure, issue_title, issue, issue_description

• Description: Issue-driven maintenance tasks: merging jobs, adding logging, improving error messages, enforcing YAML config constraints.

• Example PRs: #11060, #11077, #11084

  Example prompt (#11060): merge close issues and close discussions in same job; add extensive logging in close issues script using core.info

---

C6: Direct CI Job Failure Fixes

• Size: 33 PRs  |  Success rate: 82% ████████░░

• Top keywords: job, fix, implement, failing, analyze, root cause

• Description: Targeted prompts that supply a failing job URL + ID and ask Copilot to analyze logs, identify the root cause, and ship a fix. Highest-precision task type.

• Example PRs: #11096, #11915, #12304

  Example prompt (#11096): Fix the failing GitHub Actions workflow js. Analyze the workflow logs, identify the root cause of the failure, and implement a fix. Job ID: 61070763482 Job URL: https://github.com/githubnext/gh-aw/actions/runs/21225360006/job/61070763482

---

C7: Report Formatting Standardization

• Size: 30 PRs  |  Success rate: 87% ████████░░

• Top keywords: style, guidelines, update workflow, formatting, hierarchy, markdown

• Description: A focused campaign to normalize markdown header levels (h2→h3) and add progressive disclosure to agentic workflow reports across many workflow files.

• Example PRs: #11490, #11503, #11538

  Example prompt (#11490): [workflow-style] Normalize report formatting for auto-triage-issues — workflow uses h2 (##) headers which should be h3 (###)…

---

C8: Safe Outputs Infrastructure

• Size: 84 PRs  |  Success rate: 81% ████████░░

• Top keywords: safe, safe outputs, outputs, safe output, project, output

• Description: Tasks related to the safe-outputs MCP server container: Docker/git configuration, mounting workspace, updating base images.

• Example PRs: #11116, #11117, #11119

  Example prompt (#11116): safe-output create-pull-request is failing. Since we moved safe-outputs in its own container, it fails to invoke git properly to generate the patch that create-pull-request uses. We need to make sure that the safe-outputs has the user workspace mounted, git is properly configured…

---

Success Rate by Cluster

#	Theme	PRs	Merged	Success	Top Keywords
C1	Unclear / Meta Prompts	67	49	73%	coding agent, coding, copilot coding, agent tips
C2	Issue-Tracker–Driven Tasks	103	60	58%	task miner, miner, discussion task, discussion
C3	CI Failure Doctor – Issue-Based	176	93	53%	gh aw, aw, gh, issue_title
C4	Agentic Workflow Updates & Features	406	301	74%	agentic, update, reference, campaign
C5	Maintenance & Logging Improvements	60	38	63%	comments, failure, workflow failure, issue_title
C6	Direct CI Job Failure Fixes	33	27	82%	job, fix, implement, failing
C7	Report Formatting Standardization	30	26	87%	style, guidelines, update workflow, formatting
C8	Safe Outputs Infrastructure	84	68	81%	safe, safe outputs, outputs, safe output

Recommendations

1. Include a concrete artifact in every prompt: Clusters with a job URL, version number, or file path merge at ~80%+. Consider requiring task-miner templates to include a target artifact reference.
2. Add explicit acceptance criteria to issue-driven tasks: C2 and C3 have broad descriptions. A "Definition of Done" field in issue templates would give Copilot a clearer stopping condition.
3. Investigate C3 (CI Failure Doctor) WIP accumulation: 47% of these PRs don't merge — many may stall on flaky tests or over-broad fixes. Consider adding a max-scope hint to the CI-failure-doctor prompt.
4. Promote C6's direct job-URL pattern: The analyze the workflow logs, identify the root cause, fix it prompt with a specific job URL achieves 82% success. Use it as the canonical template for CI failure remediation.
5. C7 campaign is nearly complete: Report-formatting standardization shows 87% success with only 30 PRs — this campaign is winding down and the pattern is working well.

References: §23561258569

AI generated by Copilot Agent Prompt Clustering Analysis · history

• expires on Mar 26, 2026, 8:33 PM UTC

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-03-26T20:33:23.496Z.

Closed by Workflow