Agent Performance Report — Week of 2026-03-24

[github-actions[bot]]

Executive Summary

• Agents analyzed: 177 workflows (15 non-agentic, 162 agentic)
• Active runs reviewed: 12 completed agentic runs (7-day window)
• Estimated quality score: 82/100 (→ stable)
• Estimated effectiveness score: 76/100 (↑1 from 75)
• Top performers: Agent Container Smoke Test, Issue Monster, Contribution Check, CLI Version Checker, Documentation Unbloat
• Needs improvement: Smoke Update Cross-Repo PR (P1), Documentation Unbloat (cost efficiency)
• Stale lock files: 19 (→ stable, systemic churn)

---

Performance Rankings

Top Performing Agents 🏆

1. Agent Container Smoke Test (Reliability: ~100%)

   • 3/3 recent runs successful (runs Fix regex escaping #952, [docs] Documentation Accessibility Analysis - Automated Dev Workflow Report #953, [docs] 📋 Documentation Build and Accessibility Analysis Report #954)
   • Consistent performance: 3.5–3.7m duration, 4–7 turns per run
   • 181K tokens max (run [docs] 📋 Documentation Build and Accessibility Analysis Report #954) — efficient for scope
   • Clean safe output on every run

2. Issue Monster (Reliability: ~100% recent)

   • One transient failure observed in prior 2-week window, otherwise healthy
   • Consistent schedule adherence

3. CLI Version Checker (Reliability: 100% in window)

   • 2/2 recent runs succeeded (§23471151414, §23472626029)
   • Efficient: 4.4–5.7m, 14–25 turns, using GitHub MCP properly
   • Creates targeted issues with good tooling (github + safeoutputs)

4. Documentation Unbloat (Quality: ✅ but Cost ⚠️)

   • §23471703194: 12.6m, 37 turns, 2.7M tokens, $1.38
   • Creates PRs with real doc changes (playwright + mcpscripts + safeoutputs)
   • Heavy resource user — see optimization recommendation below

5. PR Triage Agent (10/10 schedule — FULLY RECOVERED 🎉)

   • Solidified recovery from prior extended failure
   • Now a model of resilience

6. Contribution Check (89% — stable)

   • Consistent performance across PR events

View Additional Healthy Agents

• Auto-Triage Issues: 6/6 schedule ✅
• Metrics Collector: 8/8 schedule ✅ (providing ecosystem metrics)
• Smoke Gemini: SOLIDIFIED 5/5 consecutive ✅
• Smoke Copilot: 88% (2552 runs baseline — very active)
• Smoke Codex: 87% stable
• AI Moderator: success (8.8m, clean run in prior week)
• Duplicate Code Detector: success with safe outputs
• Safe Output Health Monitor: success (11.9m)
• Grumpy Code Reviewer: success (15s on PR events — efficient)
• CI Failure Doctor: success (10s — very efficient PR event handler)

---

Agents Needing Improvement 📉

1. Smoke Update Cross-Repo PR (P1 — 9+ days continuous failure)

   • 0% success over 10+ scheduled runs
   • Issue #22241 exists and has been updated
   • Root cause: unknown — likely repo permission or cross-repo PR API issue
   • Recommendation: Investigate permissions and cross-repo token scoping; consider adding diagnostic step

2. Documentation Unbloat (Cost efficiency concern)

   • 2.7M tokens per run, $1.38/run — highest cost in ecosystem
   • 37 turns with playwright + mcpscripts suggests complex workflow but potential over-processing
   • Recommendation: Add caching check to skip runs when no doc changes needed; reduce playwright usage to targeted verification only

3. GitHub Remote MCP Authentication Test (Missing tool report)

   • §23472562940: 1M tokens, 28 turns — heavy
   • Reported missing_tool: "GitHub MCP tools (list_issues, get_repository) — MCP toolsets unavailable in runner"
   • Recommendation: Verify MCP toolset configuration; add fallback when GitHub MCP unavailable

4. Smoke Claude (Transient failure cluster — prior week)

   • 3 failures observed in Mar 17–21 window (short runs, error_count=1 each)
   • Most recent available data shows in_progress — monitor for recovery
   • Recommendation: Monitor closely; if failures persist post-Mar 24, investigate Claude API availability

---

Recovering Agents ↗️

• Issue Triage Agent: 1 success after 5+ consecutive failures. No runs Mar 21–22. Fragile recovery — watch closely.
• Daily Rendering Scripts Verifier: 2/2 holding — upgrade from recovering to stable if 3rd consecutive succeeds.

---

Quality Analysis

View Quality Breakdown

Output Quality Distribution (estimated)

• Excellent (80–100): ~12 agents (Issue Monster, Agent Container Smoke Test, PR Triage, CLI Version Checker, Metrics Collector, Auto-Triage, Smoke Gemini, Contribution Check, AI Moderator, CI Failure Doctor, Grumpy Code Reviewer, Changeset Generator)
• Good (60–79): ~8 agents (Smoke Copilot, Smoke Codex, Smoke Claude, Documentation Unbloat, Safe Output Health Monitor, Issue Triage Agent, Daily Rendering Verifier, jsweep)
• Fair (40–59): ~1 agent (GitHub Remote MCP Auth Test — missing tools)
• Poor (<40): 1 agent (Smoke Update Cross-Repo PR — 0% completion)

Common Quality Issues

1. Stale lock files (19 workflows): Agents editing .md source files without running make recompile — causes compiled lock files to diverge from source. Affects CI validity.
2. Missing tool reports: 1 instance (GitHub Remote MCP Auth Test) — MCP toolset not loaded in runner context
3. Resource over-use: Documentation Unbloat and GitHub Remote MCP Auth Test consuming disproportionate tokens per run

---

Effectiveness Analysis

View Effectiveness Metrics

Task Completion Rates

• High completion (>80%): ~14 agents
• Medium completion (50–80%): ~5 agents (Smoke Claude, Issue Triage Agent recovering)
• Low completion (<50%): 1 agent (Smoke Update Cross-Repo PR — 0%)

Resource Efficiency (7-day window)

Agent	Duration	Tokens	Turns	Cost
CI Failure Doctor	10s	~minimal	-	~$0
Grumpy Code Reviewer	15s	~minimal	-	~$0
Agent Container Smoke Test	3.5–3.7m	145–181K	4–7	~$0
CLI Version Checker	4.4–5.7m	612–1384K	14–25	~$0.49–$0.92
GitHub Remote MCP Auth Test	4.9m	1M	28	~$0 (copilot)
Documentation Unbloat	12.6m	2.7M	37	$1.38

Safe Output Usage (7-day copilot + claude)

• Total safe items created: 7 (4 copilot + 3 claude)
• Safe output types: add_comment (3), create_discussion (1), create_issue (3), create_pull_request (1), missing_tool (1)
• Appropriate usage observed — no over-creation pattern

---

Behavioral Patterns

Productive Patterns ✅

• CLI Version Checker: Efficient GitHub MCP + version comparison + targeted issue creation — textbook quality
• Agent Container Smoke Test: Consistent, bounded, deterministic — model smoke test
• Meta-Orchestrator Coordination: Campaign Manager, Workflow Health, Agent Performance sharing memory — effective coordination reducing redundant work
• PR Triage + Contribution Check: Good PR-event pipeline coverage

Problematic Patterns ⚠️

• Recompile gap: Agents editing .md workflow files without running make recompile → 19 stale .lock.yml files (different set each run = active churn)
• Documentation Unbloat over-processing: 37 turns + playwright per run suggests unconstrained analysis; may benefit from scope limiting
• Double-trigger: CLI Version Checker ran twice on 2026-03-24 (schedule + workflow_dispatch) — expected but watch for cost accumulation

---

Coverage Analysis

View Coverage Map

Well-Covered Areas ✅

• Smoke testing (Copilot, Claude, Codex, Gemini, ARM64, Agent containers)
• PR review and triage
• Issue management and triage
• CI health monitoring
• Documentation maintenance
• Dependency and version tracking
• Schema consistency

Coverage Gaps (potential opportunities)

• Security vulnerability tracking (no dedicated security-focused agent visible in this week's active runs)
• Performance regression detection (no perf monitoring agent active in window)
• Test coverage analysis

Redundancy

• 4 smoke engines (Copilot/Claude/Codex/Gemini) + specialized smokes (ARM64, Project, Cross-Repo PR, Temporary ID, Call Workflow, Multi PR) — comprehensive coverage intentionally
• 2 PR analysis agents (Grumpy Code Reviewer + Contribution Check) — complementary, not redundant

---

Recommendations

High Priority

1. Fix Smoke Update Cross-Repo PR (P1 — 9+ consecutive failures)

   • Issue #22241 already exists
   • Next step: debugging workflow run to identify exact failure mode
   • If no fix by March 30, consider temporarily disabling to reduce noise

2. Investigate GitHub Remote MCP Auth Test MCP toolset loading

   • Add explicit toolsets: [default] or verify runner MCP config
   • Consider adding graceful degradation when GitHub MCP unavailable
   • Reduce turn budget given 28 turns for an auth test seems excessive

3. Address Documentation Unbloat cost efficiency

   • Add cache-memory check at start: skip run if no recent doc commits
   • Limit playwright usage to targeted validation only
   • Set explicit turn/token budget in prompt
   • Target: reduce from $1.38/run to under $0.50

Medium Priority

4. Stale lock file prevention (19 ongoing)

   • Consider adding a CI check that flags agents creating PRs without recompiling
   • Add reminder to workflow prompts that edit .md files: "Always run make recompile after editing workflow markdown"

5. Monitor Issue Triage Agent recovery

   • If no runs by March 27, trigger manual dispatch to test
   • Check if scheduler is correctly configured post-last-fix

6. Monitor Smoke Claude stabilization

   • Prior week had 3 short failures; appears in_progress now
   • Flag if failure rate exceeds 20% over next 7 days

Low Priority

7. Document jsweep baseline — new workflow, establishing baseline; monitor first 5 runs
8. Schema Consistency Checker — first run visible; assess output quality next cycle

---

Trends (vs. previous report 2026-03-23)

Metric	Previous	Current	Trend
Quality Score	82/100	82/100	→ stable
Effectiveness Score	75/100	76/100	↑+1
Ecosystem Health	71/100	71/100	→ stable
Total Workflows	177	177	→
Stale Lock Files	19	19	→ (same count, different set)
P1 Issues	1	1	→ (Smoke Update Cross-Repo PR)
PR Triage Recovery	Holding	Holding strong	✅
Smoke Gemini	Solidified	Healthy	✅

---

Actions Taken This Run

• Generated this performance report discussion
• Updated agent-performance-latest.md in repo memory
• Updated shared-alerts.md with current coordination notes
• No new improvement issues created (existing issue [health] Smoke Update Cross-Repo PR: 100% failure rate on schedule (6/6 consecutive failures) #22241 covers P1; other items are monitoring/optimization)

Next Steps

1. Investigate Smoke Update Cross-Repo PR root cause (issue [health] Smoke Update Cross-Repo PR: 100% failure rate on schedule (6/6 consecutive failures) #22241)
2. Monitor Issue Triage Agent — trigger manual if no runs by Mar 27
3. Monitor Smoke Claude stabilization over next 7 days
4. Review Documentation Unbloat cost optimization (medium priority)
5. Assess GitHub Remote MCP Auth Test MCP configuration

---

Analysis period: 2026-03-17 to 2026-03-24
Next report: 2026-03-31
References: §23473077830 · §23426422007 · §23471703194

AI generated by Agent Performance Analyzer - Meta-Orchestrator · history

• expires on Mar 25, 2026, 4:37 AM UTC

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-03-25T04:37:33.818Z.

Closed by Workflow