Add prompt-eval-gate loop (LLMOps regression gate)

[DCCA]

Summary

Adds the prompt-eval-gate loop (rank #4 from deep-dive #3) — an LLMOps regression gate over a prompt eval set, on the durable StateStore + a human Gate. loopy dogfoods it on its own AI loops' prompts.

On a pull_request it runs the eval set through the model, grades deterministically, and compares to a baseline in the StateStore:

• no baseline → establish it (comment);
• regression (a case that passed in baseline now fails) → blocking advisory comment, baseline unchanged;
• improvement → human-gated promotion: a comment until approved, then a PR writing evals/baseline.json + persisting the new baseline.

Uses both output channels (comment + PR). The CLI builds the model from the OpenRouter AI client, reads cases from LOOPY_EVAL_CASES_FILE, file-backed state; run() now resolves the PR number for event-triggered loops so comment output works.

OpenSpec

Source of truth: openspec/specs/prompt-eval-gate.md.

Validation

• typecheck / lint — clean
• npm test — 172 tests (+6: establish / regress / gated-promote / stable + helpers)
• npm run build — clean

Next from the ranking: model-upgrade-migration, then api-deprecation-rollout.

🤖 Generated with Claude Code

https://claude.ai/code/session_012CrMioorha3QtPUgWbtDbR

---

Generated by Claude Code

[DCCA]

Code review

CI green (build ~20s); locally typecheck, lint, 172 tests, build all pass.

Strengths

• Clean baseline-gate semantics, each covered by a test: no baseline → establish + persist; regression → blocking comment with the baseline never moved; improvement → human-gated, comment until approved, then a PR writing evals/baseline.json + persisting state; stable → no-work.
• Right primitives: durable baseline in the StateStore, promotion behind a Gate, and it exercises both output channels (comment + PR) — the dual-output path is correctly driven by the runner's outputKind.
• Deterministic grading (substring + tolerance) sidesteps flaky LLM-judged thresholds; the model/eval-set are injected, so it's fully unit-tested with fakes.
• The run() change to resolve the PR number for any event-triggered loop (not just output: "comment") is the correct fix so comment output works here.

Notes (non-blocking)

• detect and act both run the eval (the model boundary) — fine with the fake; for a real model this is 2× calls per run. An optimization (thread the scorecard from detect) is a reasonable follow-up; consistent with how other loops recompute in act.
• Substring grading is intentionally simple; rubric/per-category scorecards are noted future work.

Recommend merge.

---

Generated by Claude Code