(MOT-4168) harness: prompt eval runner for live-rig conduct benching

[rohitg00]

What

A committed, zero-dependency prompt eval runner at harness/evals/prompts: run.mjs + scenarios.json + README. It benches system-prompt variants against a live rig and grades agent conduct, not exact text.

• Arms: current sends no override (whatever the router serves the rig, token count fetched live via router::system_prompt::get), candidate sends harness/prompts/default.txt from the checkout with system_prompt_strategy: override. Any prompt file can be benched by pointing an arm at it.
• Assertions per scenario: result_matches, calls_include, calls_exclude, call_order (first match of before precedes first match of after), arm-scoped, with an optional flag for observed-but-not-gating checks.
• Shipped matrix covers the conduct that prompt changes actually regress: no orchestration machinery on simple tasks, playbook pull before spawning, bulk payloads moved with fp::pipe instead of flooding the context (a 1 MB fetch read inline costs roughly 270k tokens), no installs on a read-only registry question.
• Reports land in out/ (gitignored) as markdown + JSON with per-arm prompt tokens, per-session call sequences, turns, and generations. Exit 1 on any non-optional failure.

Why

harness/evals/integration is the deterministic conformance gate: isolated stack, scripted router::*, exact invariants. A scripted router cannot measure prompt-driven behavior, which is exactly what changes when a prompt changes. This runner is the live-model complement, and the working seed of the HarnessBench tech spec (matrix runs comparing prompt and config variants). The methodology is the one that validated #558/#560: 13 sessions of ad hoc scripts, now committed and repeatable.

Proof run (live rig, engine 0.21.6, claude-sonnet-5)

| scenario        | arm       | pass | turns | gens | calls |
| simple-ls       | current   | pass | 4     | 4    | 3     |
| simple-ls       | candidate | pass | 4     | 4    | 3     |
| bulk-two-fields | current   | pass | 4     | 4    | 7     |
| bulk-two-fields | candidate | pass | 3     | 3    | 3     |

Both arms piped the 1 MB payload (web::fetch -> fp::get -> fp::size inside one fp::pipe) with zero body tokens in the model context. On this branch candidate is the current 8,799-token default.txt; once #558 lands the same run shows the drop to 1,676.

Notes

• Requires a running engine + harness + real provider + iii CLI; not wired into CI on purpose (live model, nondeterministic text, deliberate cost).
• Sessions are named pbench-<runid>-<arm>-<scenario> and stay inspectable in the console.

Linear

Closes MOT-4168. Part of MOT-4157.

[coderabbitai]

Warning

Review limit reached

@rohitg00, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 58 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: e88f0bff-84fd-494b-ac97-8d8ee3007786

📥 Commits

Reviewing files that changed from the base of the PR and between 28f44dc and 7a476dd.

📒 Files selected for processing (4)

• harness/evals/prompts/.gitignore
• harness/evals/prompts/README.md
• harness/evals/prompts/run.mjs
• harness/evals/prompts/scenarios.json

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch harness-prompt-evals

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
workers	Ready	Preview, Comment	Jul 22, 2026 4:13pm
workers-tech-spec	Ready	Preview, Comment	Jul 22, 2026 4:13pm

[github-actions]

skill-check — worker

0 verified, 47 skipped (no docs/).

Layer	Result
structure	✓
vale	✓
ai	✓
render	✓

Four for four. Nicely done.