🏥 Repository Health Dashboard

[github-actions]

🏥 Daily Health Check — 2026-06-21

Status: 🔴 1 critical · 🟡 1 warning · 🔵 1 info
Since last check (2026-06-20): 🆕 0 new · ✅ 2 resolved · 📌 3 unchanged

📌 Maintainer action needed: please pin this issue as the canonical health dashboard and unpin/close any stale duplicate.

---

🆕 New Findings (0)

No new findings detected since the last health check (2026-06-20).

---

🔍 Investigation Results

Deep investigations are dispatched for new critical/warning findings.
The grooming workflow links results ~3 hours after this run.

Finding	Severity	Investigation	First Seen	Result
Evaluation avg duration critical (110+ min)	🔴 Critical	✅ Done	2026-06-03	The evaluate (dotnet-test) job is the sole critical bottleneck at 100 minutes, driven by 175 sequential eval scenarios; the regression traces directly to PR #707 (merged June 1) which added polyglot scenarios to three skills.

---

✅ Resolved Since Last Check (2)

These were in the previous report but are no longer detected.
⚠️ Note: cache was last saved 2026-06-13 (Jun 20 run did not persist cache); these findings were already shown as resolved in the 2026-06-20 dashboard.

• 🔴 Critical — Evaluation failed: evaluate (dotnet-diag) — Run skill-validator (pipeline:evaluation:evaluate-(dotnet-diag):run-skill-validator:failure)
  All scheduled evaluation runs since Jun 13 have completed successfully. The HTTP 429 token rate-limit that caused the dotnet-diag failure has cleared. Latest scheduled run #4514 (Jun 21) succeeded.

• 🔴 Critical — Evaluation failure rate critical (50% in last 24h) (pipeline:evaluation:failure-rate:critical)
  Current failure rate in the last 24h: 0% (0 failures / 3 non-skipped runs). The spike on 2026-06-13 was transient and has fully self-resolved.

---

📌 Existing Findings (3)

These have been present since before today. Sorted by severity then age.

🔴 Critical — Evaluation average duration critical (~145 min avg, threshold: 55 min) · first seen 2026-06-03 · 9 occurrences

• Fingerprint: resource:eval-duration:critical
• First seen: 2026-06-03 · Occurrences: 9 (chronic — 7+ weeks)
• Details: The 14-day average for substantial evaluation runs is ~145 min — over 2.6× the 55-min critical threshold. The most recent scheduled evaluation on main (Jun 19) ran 164 min; the Jun 19 issue_comment-triggered runs clocked 139 min and 132 min respectively. Jun 20 and Jun 21 scheduled runs were no-ops (no new commits since last eval).
• 7-day run summary: 3 substantial runs (132, 139, 164 min) + 2 no-op schedule runs + 3 skipped issue_comment runs. 1 failure (build-validator) on Jun 19, now resolved.
• Root cause (from investigation): The evaluate (dotnet-test) job is the bottleneck (~100+ min), driven by 175 sequential eval scenarios added by PR Make dotnet-test analysis skills and auditor agent polyglot #707 (merged 2026-06-01).
• Links: Latest scheduled run (Jun 19) · Investigation result · evaluation.yml
• Suggested action: Parallelize eval scenarios in evaluate (dotnet-test); consider splitting 175 sequential scenarios across multiple parallel jobs, or introduce a fast-path for PR-triggered evaluations vs. nightly scheduled runs.

🟡 Warning — Orphan plugin: dotnet-experimental not listed in marketplace.json · first seen 2026-05-14 · 28 occurrences

• Fingerprint: infra:orphan-plugin:dotnet-experimental
• First seen: 2026-05-14 · Occurrences: 28 (chronic — 5+ weeks)
• Details: The directory plugins/dotnet-experimental/ exists on disk with a valid plugin.json (name: dotnet-experimental, version: 0.1.0) and a skills/ directory containing exp-mock-usage-analysis, exp-simd-vectorization, and exp-test-maintainability. However, no corresponding entry exists in .github/plugin/marketplace.json (which lists 13 plugins).
• Links: marketplace.json · plugins/dotnet-experimental
• Suggested action: Add { "name": "dotnet-experimental", "source": "./plugins/dotnet-experimental", "description": "..." } to marketplace.json when ready to publish, or remove the plugin directory if not intended for publication.

🔵 Info — evaluation.yml uses --verdict-warn-only mode · first seen 2026-05-16 · 26 occurrences

• Fingerprint: infra:verdict-warn-only
• Details: evaluation.yml passes --verdict-warn-only to the skill-validator, meaning skill validation failures are treated as warnings rather than hard failures. This is intentional configuration.
• Link: evaluation.yml

---

📊 Trends (7-day)

Metric	Today	7d Avg	Δ	Trend
Eval duration — scheduled main (min)	0 (no-op; last actual: 164 Jun 19)	~145 (substantial runs)	—	➡️ Stable
Eval success rate — main (7d non-skipped)	83% (5/6)	~83%	0%	➡️ Stable
Eval success rate — all branches (7d non-skipped)	94% (15/16)	~94%	0%	➡️ Stable
Eval scheduled cancellation rate (24h)	0% (0/1 scheduled)	~0%	0%	➡️ Stable
Workflow failure rate — main (24h)	0%	Low	—	✅ Improving
Compute hours/day	~0.2h (no eval work today)	~4h	-3.8h	✅ Improving

✅ Pipeline fully healthy today: no failures, no cancellations, 0% failure rate in last 24h on all branches.
⚠️ Eval duration remains a chronic concern (~145 min avg when running). No actual evaluation ran Jun 20–21 (no-ops due to no new commits).
i️ Cache note: last saved 2026-06-13; Jun 20 run updated the issue but did not persist cache. 2 resolved findings from Jun 13 are shown as resolved again (already resolved in the 2026-06-20 dashboard).
⚠️ Skipped I5 check (Pages deployment): GitHub Pages API not accessible via available tools.
i️ I6 (unpinned actions): Enforce Action Pinning workflow is active; no manual scan performed today.

---

_{🤖 Generated by DevOps Health Check agentic workflow · Run #27893641369 · 2026-06-21T04:39 UTC}

Generated by DevOps Daily Health Check · sonnet46 4.9M · ◷

[github-actions]

🔍 Investigation: Evaluation average duration critical (110+ min)

Finding ID: resource:eval-duration:critical
Severity: Critical
Correlation: hc-2026-06-03-1
Executive Summary: The evaluate (dotnet-test) job is the sole critical bottleneck at 100 minutes, driven by 175 sequential eval scenarios; the regression traces directly to PR #707 (merged June 1) which added polyglot scenarios to three skills.

Root Cause

The evaluate (dotnet-test) job took 100 minutes in run #3963 (skill-validator step: 01:11:21Z → 02:51:30Z), making it the last job to unblock post-processing and the primary driver of total run duration. A full count of tests/dotnet-test/**/eval.yaml shows 175 scenarios running sequentially in a single job. The prior scheduled run (June 2, 107.7 min) started the same trend, confirming this is not a transient spike.

The growth directly follows PR #707 (680292e, merged 2026-06-01 14:46Z — "Make dotnet-test analysis skills and auditor agent polyglot"), which explicitly added new polyglot eval scenarios to three skills:

• test-anti-patterns — 7 scenarios (Python/pytest, TS/Jest, Java/JUnit added alongside C#)
• assertion-quality — 6 scenarios (polyglot scenarios added)
• test-smell-detection — 5 scenarios (polyglot scenarios added)

A secondary bottleneck is evaluate (dotnet-msbuild--shard-medium) at 79 minutes (parallel, so it doesn't add wall time on top of dotnet-test, but it does drive high compute usage across all 3 msbuild shards).

Job duration summary for run #3963 (all parallel from ~01:11):

Job	Duration
evaluate (dotnet-test) ← bottleneck	100 min
evaluate (dotnet-msbuild--shard-medium)	79 min
evaluate (dotnet-ai)	71 min
evaluate (dotnet-msbuild--shard-default)	64 min
evaluate (dotnet-blazor)	58 min
evaluate (dotnet-msbuild--shard-heavy)	54 min
evaluate (dotnet-upgrade)	47 min
evaluate (dotnet-diag)	40 min

Confidence: High — direct evidence from job timestamps; PR #707 commit message explicitly states "New polyglot eval scenarios added ... in test-anti-patterns, assertion-quality, test-smell-detection"; scenario count (175) corroborated by direct file count.

Blast Radius

• Every scheduled evaluation run exceeds the 55-min critical threshold; the 7-day trend is ↗️ Increasing (~120 min average).
• Feedback latency for PR authors using /evaluate is now ~2.5× the expected duration.
• If the schedule interval is ≤2 hours, cascading run cancellations are possible (the health check flagged this as P6 risk).
• The secondary msbuild bottleneck (3 shards, up to 79 min) adds significant compute cost in parallel.

Suggested Fix

1. Shard evaluate (dotnet-test) — dotnet-msbuild already splits its 3 shards (heavy/medium/default); apply the same pattern to dotnet-test. With 175 scenarios, 3–4 parallel shards would reduce the job to ~25–35 min. Group the heavy agentic scenarios (code-testing-agent, agent.test-*) in one dedicated shard and distribute the simpler analysis skills across the others.

2. Review the scenario density added by PR Make dotnet-test analysis skills and auditor agent polyglot #707 — The three newly-polyglot skills contributed ~10–15 new scenarios. If each scenario averages ~30–35 seconds of runtime, consolidating or reducing these could recover 5–9 minutes. Consider whether Python/TS/Java coverage can share fixtures or prompts to reduce scenario count without losing coverage breadth.

3. Reduce dotnet-msbuild--shard-medium runtime (79 min) — Audit which scenarios in the medium shard are slowest and consider whether any can be promoted to the heavy shard or simplified.

Evidence

# Job: evaluate (dotnet-test) — run #3963
# started_at: 2026-06-03T01:11:04Z
# completed_at: 2026-06-03T02:51:46Z  (100 min)
# Bottleneck step: Run skill-validator  01:11:21Z → 02:51:30Z

# dotnet-test eval scenario count (tests/dotnet-test/**/eval.yaml):
run-tests: 15, writing-mstest-tests: 15, migrate-mstest-v3-to-v4: 15,
migrate-vstest-to-mtp: 11, dotnet-test-frameworks: 11, migrate-xunit-to-xunit-v3: 12,
test-tagging: 8, mtp-hot-reload: 7, test-anti-patterns: 7, assertion-quality: 6,
detect-static-dependencies: 6, test-smell-detection: 5, migrate-mstest-v1v2-to-v3: 9,
migrate-static-to-wrapper: 3, coverage-analysis: 3, crap-score: 3,
test-gap-analysis: 4, generate-testability-wrappers: 4, agent.test-migration: 4,
agent.test-quality-auditor: 4, agent.testability-migration: 2,
code-testing-agent: 1  →  TOTAL: 175 scenarios

PR that introduced the regression:

• PR #707 — "Make dotnet-test analysis skills and auditor agent polyglot" — commit 680292e — merged 2026-06-01 by Evangelink

Related

• PR Make dotnet-test analysis skills and auditor agent polyglot #707 (680292e) — direct cause: added polyglot scenarios to test-anti-patterns, assertion-quality, test-smell-detection
• PR code-testing-agent: add polyglot pipeline examples for Python/TypeScript/Go/Java #708 (49a77a9) — coincident merge (June 2), adds documentation only, not a cause
• Trends table in this issue: 7d avg for scheduled eval = ~120 min (↗️ Increasing)
• No existing tracking issue found for the dotnet-test evaluation sharding work

---

_{🔍 Investigation Run #26864313000 · Dispatched by health check · hc-2026-06-03-1}

Generated by DevOps Health — Deep Investigation · ● 2.3M · ◷

[github-actions]

📋 Health Check — 2026-06-15

🆕 0 new · ✅ 2 resolved · 📌 3 unchanged

Status: 🔴 1 critical · 🟡 1 warning · 🔵 1 info

Resolved:

• 🔴 Evaluation failed: evaluate (dotnet-diag) — Run skill-validator — dotnet-diag token 429 issue cleared after June 14 run succeeded
• 🔴 Evaluation failure rate critical (50% in last 24h) — failure rate is now 0% (1 run, success)

Persistent (unchanged):

• 🔴 Evaluation average duration critical (~116 min, threshold 55 min) — 9 occurrences since 2026-06-03 · PR #707 bottleneck still unaddressed
• 🟡 Orphan plugin: dotnet-experimental not in marketplace.json — 28 occurrences since 2026-05-14
• 🔵 evaluation.yml uses --verdict-warn-only — intentional config

No investigation dispatches — no new qualifying findings this run.

Full report →

Generated by DevOps Daily Health Check · sonnet46 6.2M · ◷

[github-actions]

📋 Health Check — 2026-06-20

🆕 1 new · ✅ 2 resolved · 📌 3 unchanged

Status: 🔴 2 critical · 🟡 1 warning · 🔵 1 info

New:

• 🔴 Evaluation failed: build-validator — Build skill-validator — build-validator job failed at "Build skill-validator" step on main (run #4504, issue_comment trigger). All evaluate jobs skipped. Investigation dispatched.

Resolved:

• 🔴 Evaluation failed: evaluate (dotnet-diag) — Run skill-validator — token rate-limit cleared, Jun 19 scheduled run succeeded.
• 🔴 Evaluation failure rate critical (50% in last 24h) — failure rate now 8.3% (well below 15% threshold).

Notable (existing):

• 🔴 Eval duration still critical: 164 min on Jun 19 (up from 116 min on Jun 14, PR Make dotnet-test analysis skills and auditor agent polyglot #707 bottleneck unaddressed)

Full report →

Generated by DevOps Daily Health Check · sonnet46 7.3M · ◷

[github-actions]

📋 Health Check — 2026-06-21

🆕 0 new · ✅ 2 resolved · 📌 3 unchanged

New: (none — pipeline fully healthy today)

Resolved (from cache 2026-06-13; already resolved in 2026-06-20 dashboard):

• 🔴 Evaluation failed: evaluate (dotnet-diag) — Run skill-validator — all scheduled eval runs now completing successfully
• 🔴 Evaluation failure rate critical (50% in last 24h) — current rate: 0% (0/3 non-skipped runs)

Ongoing concerns:

• 🔴 Evaluation average duration critical (~145 min avg, 7+ weeks) — no actual eval ran Jun 20–21 (no new commits)
• 🟡 Orphan plugin: dotnet-experimental (not in marketplace.json) — 28 occurrences
• 🔵 --verdict-warn-only mode (intentional)

No new dispatches: 0 qualifying new findings.

Full report →

Generated by DevOps Daily Health Check · sonnet46 4.9M · ◷