feat(evals): three-tier skill eval framework (trigger, routing, behavioral)

[addyosmani]

Why

We have 24 skills and no way to measure whether they work - well, beyond manual testing :) The validator checks structure, but nothing checks the things that matter in practice: does a skill trigger when a user asks for it in their own words, do two skills' descriptions blur into each other, and does an agent following a skill behave the way the skill promises. Every new skill PR makes this gap more expensive.

What the community has settled on (I went looking first)

Short version: there isn't one settled standard for skill evans, but there are two clear leaders.

• Anthropic's skill-creator v2 is the closest thing to a standard. It defines a per-skill evals.json (prompt plus expectations, graded from the transcript) and tooling for trigger accuracy, benchmarks, and variance analysis.
• Superpowers (obra) tests skills with bash, headless claude -p, prompt fixtures, and grader scripts. Our stale feat: introduce evaluation framework for skill testing #51 was already modeled on this, credit where due.

Neither covers what a multi-skill catalog needs deterministically in CI: routing. So the approach here is to adopt the community schema where it exists and define the missing piece ourselves.

What this adds

Three tiers, defined in evals/README.md:

1. Structural. Already existed (validate-skills.js, validate-commands.js). Unchanged.
2. Trigger and routing. New, deterministic, runs in CI. Every skill gets an eval file with realistic positive prompts (must rank the skill top-k when scored against all 24 descriptions), negative prompts (must not rank first), plus a collision check so no two descriptions drift into near-duplicates. Zero dependencies, plain node, no API calls, so it's free and fast on every PR.
3. Behavioral. New, opt-in, costs tokens, never runs in CI. Each skill ships evals in skill-creator's exact evals.json schema. node scripts/run-evals.js --behavioral <skill> runs them through headless claude -p and grades the transcript against the expectations. There's a --dry-run so you can preview without spending anything.

The schema compatibility is deliberate: because our evals[] block matches skill-creator's format field for field, you can point Anthropic's own runner, benchmarks, and eval viewer at this repo without changing a thing.

What's in the box

• 24 eval case files: 72 positive trigger prompts, 48 negative ones, 24 behavioral evals with ~80 graded expectations
• scripts/run-evals.js, a zero-dependency runner (~320 lines)
• CI wiring in the validate-skills job
• CONTRIBUTING now asks new skills to ship with an eval file (warning-level for now so we don't break the in-flight skill PRs; we can flip it to an error once those clear)

Baseline on main today: 120 checks pass, 85% of positive prompts rank their skill first, zero catalog collisions.

It already caught two real things

Writing fair prompts surfaced two description gaps I did not paper over: frontend-ui-engineering's description never says "responsive", "accessible", or "WCAG", and performance-optimization's never mentions queries or N+1. A user saying those words may not route to the right skill. I kept the eval prompts honest rather than quietly editing the descriptions here; that's a small follow-up PR, and these evals will verify it.

Coordination notes

• feat: introduce evaluation framework for skill testing #51 (eval framework proposal from April): this supersedes it for the harness, but the fixtures idea in it is still good input for tier 3. I'll comment there.
• test: add skill validation battery with node:test #324 (draft test battery): no file overlap. That PR tests the validator's internals; this one evals the skills themselves. They compose.

Test plan

• node scripts/run-evals.js passes clean (120 checks, 0 errors, 0 warnings)
• node scripts/validate-skills.js and validate-commands.js still pass
• --behavioral test-driven-development --dry-run prints the execution plan without spending tokens
• Workflow YAML parses

[federicobartoli]

Hi @addyosmani! Really strong PR.

I pulled the branch locally and verified:

• node scripts/run-evals.js → 120 checks passed, 0 errors, 0 warnings
• node scripts/validate-skills.js → passed
• node scripts/validate-commands.js → passed
• node scripts/run-evals.js --behavioral test-driven-development --dry-run → prints the expected plan

A few thoughts, mostly around making Tier 3 trustworthy before we lean on it heavily:

1. Tier 3 may need to grade an execution trace, not just the captured final/reporting output. Many expectations are process claims: “a failing test is written and shown failing before the fix”, “the full suite is run after the fix”, etc. If the grader does not see the tool-call / turn-by-turn trace, it may grade whether the model narrated the behavior rather than whether it happened. --output-format stream-json + --verbose, or another transcript source with tool calls, would make those expectations much more meaningful.

2. Behavioral evals probably need a real workspace. The schema supports files[], but the runner currently does not materialize fixtures or run Claude in a temp project. Prompts like “Fix the reported rounding bug in the invoice totals, test-first” need code to operate on; otherwise Tier 3 risks measuring whether the model can describe TDD rather than perform it. A temp dir + fixture materialization + explicit tool permissions would make this much stronger.

3. Negative trigger prompts can pass vacuously when they do not meaningfully match any skill. One possible evolution: let negatives declare the expected owner skill and assert that owner outranks this skill. That would turn negatives into real pairwise routing tests instead of only “this skill must not rank first”.

4. The README documents minimums — 3 positive triggers, 2 negative triggers, 1 behavioral eval — and the current case files satisfy them, but the runner does not enforce those counts yet. I’d consider making that a schema warning for now, then an error once the transition window closes.

5. Future ratchet idea: the rank-1 rate is useful but currently only printed. Once the baseline stabilizes, a --min-rank1 CI flag could prevent silent regression while keeping top-k as the hard gate. Related: requiring one top_k: 1 signature prompt per skill could make routing quality more legible.

6. Small Tier-3 hardening: add a Node-level timeout around execFileSync('claude', ...), parse/validate grader JSON before writing it, and fence/delimit the transcript in the grader prompt so transcript content cannot accidentally instruct the grader.

None of this blocks the deterministic tier, which is the part running in CI and looks solid. This gives the repo a real measurement spine; I’d just be careful to make Tier 3 measure actual behavior rather than polished self-reporting before using it as evidence for pressure tests.

[nucliweb]

Agree with @federicobartoli’s feedback; the deterministic tier is solid and mergeable as-is; Tier 3 needs trace + real fixtures before it’s used as evidence for anything beyond “sanity check.” A few additions:

Blocking before relying on Tier 3 results:

• Until execution traces and real fixtures land, I’d treat every existing behavioral eval case as provisional. Worth a comment/flag in the case files themselves (e.g. "trust_level": "self-reported") so nobody cites current Tier 3 pass rates as validation in the meantime.

Tracking, not blocking:

• The two skills with known description gaps (frontend-ui-engineering, performance-optimization) should get an issue now, even though the fix is deferred — otherwise they’re likely to fall through since they’re only mentioned in this PR’s description, not tracked anywhere.
• Suggest bundling the two “ratchet” ideas (rank-1 minimum, enforced case-count minimums) into a single follow-up issue rather than two separate implicit TODOs — they’re the same shape of problem (observational metric → CI gate) and easier to land together.
• The CONTRIBUTING warning-level check needs a target date or linked issue for promotion to error-level, same concern as the minimums point above — otherwise “warning for now” tends to become permanent.

None of this blocks the deterministic tier — happy to see that part land now with Tier 3 hardening tracked as explicit follow-ups rather than open-ended future work.

[nucliweb]

Verified locally (on top of Federico's checks): added a throwaway 25th skill (dummy-catalog-probe, minimal SKILL.md + matching case file) to check for fixed-count assumptions in the catalog logic.

run-evals.js scaled cleanly:

Running skill evals across 25 skills, 25 case files
125 checks passed — 0 error(s), 0 warning(s)
trigger rank-1 rate: 85% (64/75 positive prompts rank their skill first)
PASSED

No hardcoded skill count anywhere in the TF-IDF corpus build, coverage check, or collision scan — confirmed by reading loadSkills()/buildCorpus() too, not just the passing run.

Side note (not a bug, just worth flagging): validate-skills.js correctly failed on my dummy skill for missing required sections (Common Rationalizations, Red Flags, Verification) — good, that's the validator doing its job on an intentionally incomplete skill, not a regression.

Removed the dummy skill/case file after testing — not part of this diff.

[addyosmani]

This is exactly the review this PR needed, thank you both. Federico, the "grades whether the model narrated the behavior rather than whether it happened" line was the sharpest observation in the thread, and you're right that it's the difference between a measurement spine and a vibes generator. And nucliweb, spinning up a throwaway 25th skill to probe for fixed-count assumptions is above-and-beyond reviewing; glad the catalog logic held up (and yes, the validator yelling at your intentionally incomplete dummy skill was it working as intended).

Pushed two follow-up commits addressing the blocking-adjacent stuff. Tier 3 now captures the full stream-json execution trace with --verbose, so the grader judges tool calls and file edits instead of prose, with the trace fenced as untrusted data, timeouts on both calls, and grader JSON validated before it's written. Each eval runs in a throwaway workspace and files[] fixtures materialize from evals/fixtures/, so the mechanism Federico asked for is wired even though fixture content is still to be authored. Every behavioral eval now carries trust_level: "provisional" per nucliweb's suggestion, and the runner says so out loud, so nobody cites current Tier 3 numbers as evidence. On the deterministic side, negatives can declare an owner that must outrank the skill, which killed the vacuous-pass problem: 37 of 48 negatives are now real pairwise routing tests, and wiring them up immediately exposed five prompts where the would-be owner had zero lexical overlap. Federico called that failure mode before I'd seen a single instance of it. Minimums are now warning-level checks in the runner too.

The rest is tracked rather than left as vibes, exactly as nucliweb suggested: #351 covers the frontend and performance description gaps (including the one owner I had to defer), and #352 bundles the whole graduation-and-ratchet path in one place: fixture authoring per skill, dropping trust_level as they land, the --min-rank1 gate, and promoting the minimums and coverage warnings to errors once the in-flight skill PRs clear. Both are linked from the README and CONTRIBUTING so "warning for now" has an explicit expiry instead of becoming permanent. Deterministic tier still holds at 120 checks, 0 errors, 85% rank-1. Thanks again to my two most reliable reviewers; this framework is meaningfully more trustworthy than what I opened with.

[nucliweb]

Two Tier-3 findings from re-reading run-evals.js after the follow-up commits, both latent because Tier 3 hasn't run against real fixtures yet:

1. The grader passes the full trace as a CLI argument; it will hit ARG_MAX. In runBehavioral, graderPrompt embeds the entire executor trace (--output-format stream-json, up to the 64MB maxBuffer) and is handed to execFileSync('claude', ['-p', graderPrompt]) as an argv entry. Any non-trivial trace will exceed the OS argument limit (E2BIG, ~256KB on macOS, a couple MB on Linux) and throw before grading. The fix is to pipe the prompt via stdin instead of argv, e.g. execFileSync('claude', ['-p'], { input: graderPrompt, ... }). Worth flagging now because CI never exercises this path, so nothing catches it until someone authors fixtures and runs a behavioral eval for real.

2. The executor call sets no tool-permission mode. The executor claude -p runs with --verbose --output-format stream-json --append-system-prompt <SKILL.md> but no permission flag or --allowedTools. In headless mode, file edits and command runs can be denied, which means the agent narrates TDD rather than performing it, the exact failure mode the trace-grading change was meant to close. Materializing the workspace + fixtures only helps if the agent is actually allowed to touch them. This likely needs an explicit permission mode (or scoped --allowedTools) on the executor call.

Neither blocks the deterministic tier, which is solid and mergeable as-is. Both fit naturally under the #352 graduation work, before any Tier-3 numbers get cited as evidence.

[addyosmani]

Great catches @nucliweb, and slightly humbling ones: both bugs live on the exact path we hardened last round, and both would have survived every CI run precisely because CI never exercises Tier 3. The ARG_MAX one especially would have been a confusing first-contact failure for whoever authored fixtures first and ran a behavioral eval for real.

You suggested parking these under #352, but they were small enough to fix now, so they're in. The grader prompt (and the executor prompt, for symmetry) now go over stdin instead of argv, and the executor runs with --permission-mode acceptEdits plus a pre-approved tool list (Read, Glob, Grep, Edit, Write, Bash) so the agent can perform the skill in its throwaway workspace instead of being denied into narration. Both are documented in the evals README, the dry-run output reflects the real invocation now, and the deterministic tier still holds at 120 checks, 0 errors, 85% rank-1. Thanks for reading the code twice; that's twice it's paid off.

[nucliweb]

LGTM!

[addyosmani]

Thanks for all the help on reviews, folks!