Tier 3 · Skill/agent authors building evals. The
/audit-evalsskill compares an existing eval against the current state of the skill or agent it tests and reports drift; you need a targetplugin:skillorplugin:agentwith an eval underevals/.
Run /audit-evals when a skill or agent has changed and its evals may no longer describe it. A run follows stale evals faithfully: a rubric criterion for a check the skill dropped fails every run, a scaffold signal the skill no longer looks for proves nothing, and a positive prompt that a new boundary clause excludes is scored as a miss. Each looks like a target regression when it is an eval regression. The audit says which is which and names the skill that repairs each artifact. It edits nothing.
Use this skill when you need to:
- Check an eval after editing a skill's
SKILL.md, references, or dispatched agents, or an agent's definition - Explain an eval that started failing — or passing — unexpectedly
- Confirm SCIL, ACIL, or judge scores are measuring the current target before acting on them
- The eval does not exist yet — use
/write-scil-evals,/write-acil-evals,/write-skill-eval-rubric, or/write-agent-eval-rubricto create it. - You want the findings fixed — the audit reports only; it names the owning skill for each fix.
- You want to review the skill itself rather than its evals — use the plugin-building guidance for that.
Invoke the skill with a plugin:skill or plugin:agent argument:
/audit-evals r-and-d:code-review
/audit-evals r-and-d:gap-analyzer
The name is resolved as a skill first, then as an agent. If no argument is provided, the skill asks which target to audit.
A report in the conversation, led by a one-sentence verdict, with findings in three severities:
| Severity | Meaning | Example |
|---|---|---|
| Breaks scoring | A run cannot pass or fails for the wrong reason | A Presence criterion names src/db.js:42, which the scaffold no longer contains |
| Weakens scoring | The score is real but measures less than it claims | A trigger test whose prompt says "this branch" carries no scaffold |
| Coverage gap | Something the target does is never exercised | The skill now dispatches a concurrency analyst; no scaffold plants a race |
Every finding cites both sides — the target file and clause, and the eval file, criterion, prompt, or scaffold path — and ends with the fix command. The report closes with a repair order: scaffold first (criteria and prompts reference it), then rubric, then prompts.
The skill walks through a 5-step process with no interview pauses:
- Identify the target — run
scripts/collect-target-inputs.sh, which resolves the name as a skill or agent and lists its references, dispatched agents, siblings, and whether an eval exists; read all of it and build the target's current profile (description clauses, checks, output shape) - Validate the eval structurally — run
scripts/validate-eval.sh, the same load-time checks Skillwalker applies (prompt files, scaffolds, rubrics,expectarrays, identifiers) - Read the eval — every test entry, prompt file, rubric, and scaffold file, listing what each scaffold plants
- Compare — rubric against target, scaffold against target and rubric, prompts against the description's current clauses and siblings, deterministic expectations against the current output shape
- Report — verdict, structural findings, then findings by severity, then the repair order
- Rubric ↔ target — a criterion with no matching check in the current target fails every run; a current check with no criterion is a gap
- Scaffold ↔ target and rubric — a planted signal the target no longer checks for costs tokens and proves nothing; a criterion naming a file or line the scaffold lacks can never pass
- Prompts ↔ description — a positive prompt now excluded by a boundary clause, or a negative prompt now covered by a "use when" clause, is scored backwards; a sibling with no sibling prompt is untested
- Expectations ↔ output shape — a
result-containsstring the current template no longer guarantees fails every run; anllm-judgethreshold above what the surviving criteria can reach never passes
Run the fix commands in the report's repair order, then re-run the audit until it reports no findings, then run the eval:
./build/skillwalker test-run --eval {name}
./build/skillwalker test-eval- Building Skill Eval Scaffolds and Building Agent Eval Scaffolds — rebuilding a scaffold the audit found stale
- Writing Skill Eval Rubrics and Writing Agent Eval Rubrics — the rubric update flow
- Writing Skill-Call Evals and Writing Agent-Call Evals — adding or re-scoping trigger prompts
- Evals Reference — what the structural validation checks
- Skillwalker README — prerequisites, setup, and running tests
Next: Building Rubric Evals or Building SCIL Evals — the full workflows the repaired eval feeds into. Related: Test Scaffolding — how scaffolds provide project context inside the Test Sandbox.