Add prompt-eval-gate loop (LLMOps regression gate)

README.md

@@ -38,6 +38,7 @@ guardrail violation) it produces no output and never partially applies changes.
 | [`flake-quarantine`](loops/flake-quarantine/README.md) | Score flaky tests across runs (stateful) and quarantine the worst | deterministic + state | PR |
 | [`release-train`](loops/release-train/README.md) | Rolling Release PR (version + changelog) from conventional commits | deterministic | PR |
 | [`license-sbom-drift`](loops/license-sbom-drift/README.md) | Flag dependency licenses outside the allowlist | deterministic | PR |
+| [`prompt-eval-gate`](loops/prompt-eval-gate/README.md) | Regression-gate prompts vs a baseline (stateful); gated promotion | AI + state | comment / PR |
 
 Plus two **long-horizon** loops on the `longrun` primitives (durable state,
 approval gates, resumable advancement): the

loops/prompt-eval-gate/README.md

@@ -0,0 +1,28 @@
+# prompt-eval-gate loop
+
+A regression gate over a **prompt eval set**: run cases through the model, grade
+them, and compare to a baseline in the durable `StateStore`. Flags regressions on
+the PR and **human-gates** baseline promotions. loopy dogfoods this on its own
+AI loops' prompts.
+
+## How it works
+
+| Phase | What happens |
+|-------|--------------|
+| **trigger** | `pull_request` event (e.g. a prompt file changed) |
+| **detect** | Run the eval set; compare scorecard to the stored baseline |
+| **act** | no baseline → establish it · regressions → blocking comment · improvement → gated promotion |
+| **output** | A comment (scorecard); a PR writing `evals/baseline.json` only on approved promotion |
+| **guardrails** | Baseline never moves without the Gate; PR confined to `evals/**` |
+
+## Boundaries (injected, testable)
+
+- `services.model(input)` — the model (reuses loopy's OpenRouter AI client)
+- `services.evals.cases()` — the eval set (`{ id, input, expect }`)
+- `services.state` — the durable baseline + the promotion gate
+
+## Notes
+
+Grading is deterministic (output contains `expect`) to avoid flaky LLM-judged
+thresholds. Regressions are advisory (a comment), never an auto-merge block; the
+baseline is only promoted through the human gate.

loops/prompt-eval-gate/hooks/eval.ts

@@ -0,0 +1,65 @@
+/** One eval case: send `input` to the model; pass if the output contains `expect`. */
+export interface EvalCase {
+  id: string;
+  input: string;
+  /** substring the output must contain (case-insensitive) to pass */
+  expect: string;
+}
+
+export interface EvalSource {
+  cases(): Promise<EvalCase[]>;
+}
+
+/** The model boundary (reuses loopy's AI client in real runs). */
+export type Model = (input: string) => Promise<string>;
+
+export interface Scorecard {
+  score: number;
+  perCase: Record<string, boolean>;
+}
+
+export function grade(output: string, c: EvalCase): boolean {
+  return output.toLowerCase().includes(c.expect.toLowerCase());
+}
+
+/** Run every case through the model and produce a scorecard. */
+export async function runEval(cases: EvalCase[], model: Model): Promise<Scorecard> {
+  const perCase: Record<string, boolean> = {};
+  let pass = 0;
+  for (const c of cases) {
+    const ok = grade(await model(c.input), c);
+    perCase[c.id] = ok;
+    if (ok) pass++;
+  }
+  return { score: cases.length > 0 ? pass / cases.length : 1, perCase };
+}
+
+/** Cases that passed in the baseline but fail now (regressions). */
+export function regressions(baseline: Scorecard, current: Scorecard): string[] {
+  return Object.keys(baseline.perCase)
+    .filter((id) => baseline.perCase[id] === true && current.perCase[id] === false)
+    .sort();
+}
+
+function pct(score: number): string {
+  return `${Math.round(score * 100)}%`;
+}
+
+export function renderScorecard(
+  current: Scorecard,
+  baseline: Scorecard | null,
+  regressed: string[],
+  note: string,
+): string {
+  const lines = ["## 🤖 loopy prompt-eval", "", note, ""];
+  lines.push(
+    `- Score: **${pct(current.score)}**` +
+      (baseline ? ` (baseline ${pct(baseline.score)})` : " (new baseline)"),
+  );
+  if (regressed.length > 0) {
+    lines.push("", "### ❌ Regressions", "");
+    for (const id of regressed) lines.push(`- \`${id}\` passed in baseline, fails now`);
+  }
+  lines.push("", "_Advisory eval gate — review before merging. Baseline promotion is human-gated._");
+  return lines.join("\n");
+}

loops/prompt-eval-gate/index.ts

@@ -0,0 +1,132 @@
+import {
+  createGate,
+  type ActResult,
+  type DetectResult,
+  type Guardrails,
+  type Loop,
+  type LoopManifest,
+  type RunContext,
+  type StateStore,
+  type Trigger,
+} from "../../src/core/index.js";
+import {
+  regressions,
+  renderScorecard,
+  runEval,
+  type EvalSource,
+  type Model,
+  type Scorecard,
+} from "./hooks/eval.js";
+
+export type { EvalCase, EvalSource, Model, Scorecard } from "./hooks/eval.js";
+export { grade, runEval, regressions, renderScorecard } from "./hooks/eval.js";
+
+export interface PromptEvalConfig {
+  /** file the promoted baseline scorecard is written to */
+  baselinePath: string;
+  /** minimum score improvement (over baseline) to propose a promotion */
+  tolerance: number;
+}
+
+export interface PromptEvalServices {
+  model: Model;
+  evals: EvalSource;
+  state: StateStore;
+}
+
+const KEY = "prompt-eval:baseline";
+const PROMOTE_GATE = "prompt-eval:promote";
+
+/**
+ * Regression gate over a prompt eval set. Compares the current scorecard to a
+ * baseline stored in the StateStore:
+ * - no baseline → establish one (comment);
+ * - regressions → blocking advisory comment (baseline unchanged);
+ * - improvement → human-gated baseline promotion (comment until approved, then a PR
+ *   writing the baseline file + persisting it to state).
+ */
+export function createPromptEvalLoop(
+  config: PromptEvalConfig,
+  guardrails: Guardrails,
+  trigger: Trigger,
+  services: PromptEvalServices,
+): Loop {
+  const gate = createGate(services.state);
+
+  async function evaluate(): Promise<{ current: Scorecard; baseline: Scorecard | null }> {
+    const current = await runEval(await services.evals.cases(), services.model);
+    const baseline = await services.state.load<Scorecard>(KEY);
+    return { current, baseline };
+  }
+
+  return {
+    id: "prompt-eval-gate",
+    trigger,
+    guardrails,
+
+    async detect(ctx: RunContext): Promise<DetectResult> {
+      void ctx;
+      const { current, baseline } = await evaluate();
+      if (!baseline) return { workNeeded: true, reason: "no baseline yet" };
+      const reg = regressions(baseline, current);
+      const improved = current.score > baseline.score + config.tolerance;
+      if (reg.length > 0) return { workNeeded: true, reason: `${reg.length} regression(s)`, affected: reg };
+      if (improved) return { workNeeded: true, reason: "score improved" };
+      return { workNeeded: false, reason: "no regressions; within tolerance" };
+    },
+
+    async act(ctx: RunContext): Promise<ActResult> {
+      void ctx;
+      const { current, baseline } = await evaluate();
+
+      // First run: establish the baseline.
+      if (!baseline) {
+        await services.state.save(KEY, current);
+        return {
+          comment: renderScorecard(current, null, [], "Baseline established."),
+          summary: "prompt-eval baseline established",
+        };
+      }
+
+      // Regressions block (advisory); never move the baseline.
+      const reg = regressions(baseline, current);
+      if (reg.length > 0) {
+        return {
+          comment: renderScorecard(current, baseline, reg, "⚠️ Prompt eval regressed."),
+          summary: `prompt-eval: ${reg.length} regression(s)`,
+        };
+      }
+
+      // Improvement: gate the baseline promotion.
+      const req = await gate.require(
+        PROMOTE_GATE,
+        `Promote prompt-eval baseline to score ${Math.round(current.score * 100)}%.`,
+      );
+      if (req.status !== "approved") {
+        const note =
+          req.status === "rejected"
+            ? "Baseline promotion rejected; keeping the current baseline."
+            : "Improvement detected — approve the promotion gate to update the baseline.";
+        return { comment: renderScorecard(current, baseline, [], note), summary: "prompt-eval improvement pending" };
+      }
+
+      await services.state.save(KEY, current);
+      return {
+        changes: [{ path: config.baselinePath, op: "write", contents: JSON.stringify(current, null, 2) + "\n" }],
+        summary: "prompt-eval baseline promoted",
+      };
+    },
+  };
+}
+
+export function createPromptEvalLoopFromManifest(
+  manifest: LoopManifest,
+  services: PromptEvalServices,
+): Loop {
+  const c = manifest.config;
+  const config: PromptEvalConfig = {
+    baselinePath: typeof c["baselinePath"] === "string" ? c["baselinePath"] : "evals/baseline.json",
+    tolerance: typeof c["tolerance"] === "number" ? c["tolerance"] : 0,
+  };
+  return createPromptEvalLoop(config, manifest.guardrails, manifest.trigger, services);
+}

loops/prompt-eval-gate/loop.yaml

@@ -0,0 +1,16 @@
+id: prompt-eval-gate
+
+trigger:
+  type: event
+  events:
+    - pull_request
+
+guardrails:
+  pathAllowlist:
+    - "evals/**"
+  maxFiles: 5
+  skipIfOpenPr: true
+
+config:
+  baselinePath: "evals/baseline.json"
+  tolerance: 0

loops/prompt-eval-gate/playbook.md

@@ -0,0 +1,25 @@
+# Prompt-Eval Gate Playbook
+
+A promptfoo-style **regression gate** over a prompt eval set. Runs each case
+through the model, grades deterministically, and compares the scorecard to a
+baseline stored in the durable `StateStore`. The model and the eval set are
+injected boundaries.
+
+## Behavior
+
+1. Run every case → output → deterministic grade (output contains `expect`).
+2. Compare the scorecard to the stored baseline:
+   - **no baseline** → establish it (comment);
+   - **regressions** (a case that passed in baseline now fails) → blocking
+     advisory comment; the baseline is **not** moved;
+   - **improvement** → a human **Gate** must approve promotion; until then a
+     comment notes it; on approval, a PR writes the baseline file and the new
+     baseline is persisted.
+
+## Guardrails
+
+- The baseline is **never silently moved** — promotion is human-gated.
+- Grading is deterministic; prefer crisp `expect` assertions over fuzzy rubrics
+  to avoid flaky LLM-graded thresholds.
+- Output is advisory (a comment) except the gated baseline-promotion PR, which is
+  confined to the `evals/**` allowlist.

openspec/archive/add-prompt-eval-gate/proposal.md

@@ -0,0 +1,57 @@
+# Proposal: Prompt-Eval Gate Loop
+
+**Change ID:** `add-prompt-eval-gate`
+**Created:** 2026-06-23
+**Status:** Implementation Complete
+**Completed:** 2026-06-23
+
+---
+
+## Problem Statement
+
+Prompt/model changes can silently regress quality. An LLMOps regression gate that
+scores an eval set against a stored baseline catches it. Ranked #4 in the deep
+dive; loopy dogfoods it on its own AI loops' prompts.
+
+## Proposed Solution
+
+Add `loops/prompt-eval-gate/`: run an eval set through the model, grade
+deterministically, and compare to a baseline in the durable `StateStore`. No
+baseline → establish; regressions → blocking advisory comment (baseline
+unchanged); improvement → human-`Gate`d baseline promotion (comment until
+approved, then a PR writing the baseline file). Reuses the AI client + comment
+output channel.
+
+## Scope
+
+### In Scope
+- `loops/prompt-eval-gate/` (hooks/eval.ts, index.ts, loop.yaml, playbook, README)
+- StateStore baseline + promotion Gate; deterministic grading
+- Catalog entry + `loopy run prompt-eval-gate` (model = AI client, cases from `LOOPY_EVAL_CASES_FILE`, file state)
+- Unit tests + package export
+
+### Out of Scope
+- Rubric/LLM-judged grading (deterministic substring grading for v1)
+- Per-category risk scorecards (future)
+
+## Success Criteria
+
+- [x] No baseline → establishes one (comment) and persists it.
+- [x] Regression → blocking comment; baseline not moved.
+- [x] Improvement → gated; on approval, a PR writes `evals/baseline.json` + state updated.
+- [x] Stable within tolerance → no work.
+
+## Risks & Mitigations
+
+| Risk | Prob | Impact | Mitigation |
+|------|------|--------|------------|
+| Flaky LLM grading | Med | Med | Deterministic substring grading + tolerance |
+| Silent baseline drift | Low | High | Promotion is human-gated; regressions never move it |
+
+---
+
+## Archive Information
+
+**Archived:** 2026-06-23
+**Outcome:** Successfully implemented
+**Verification:** typecheck + lint + 172 tests + build all passing

openspec/archive/add-prompt-eval-gate/specs/prompt-eval-gate_delta.md

@@ -0,0 +1,43 @@
+# Delta: Prompt-Eval Gate Loop
+
+**Change ID:** `add-prompt-eval-gate`
+**Affects:** `loops/prompt-eval-gate/`, CLI catalog/run
+
+## ADDED
+## Requirements
+
+### Requirement: Baseline Regression Gate
+
+The prompt-eval-gate loop scores an eval set through the model and compares it to
+a baseline stored in the durable StateStore.
+
+#### Scenario: Establish baseline
+- GIVEN no stored baseline
+- WHEN the loop runs
+- THEN it records the current scorecard as the baseline and posts a comment
+
+#### Scenario: Regression blocks
+- GIVEN a case that passed in the baseline now fails
+- WHEN the loop runs
+- THEN it posts a blocking advisory comment and does not change the baseline
+
+#### Scenario: Stable
+- GIVEN no regressions and no improvement beyond tolerance
+- WHEN the loop runs
+- THEN it reports no work
+
+---
+
+### Requirement: Human-Gated Baseline Promotion
+
+An improved scorecard promotes the baseline only after human approval.
+
+#### Scenario: Improvement gated
+- GIVEN the score improved beyond tolerance
+- WHEN the loop runs without an approval
+- THEN it posts a comment noting the improvement and leaves the baseline unchanged
+
+#### Scenario: Promotion on approval
+- GIVEN the promotion gate is approved
+- WHEN the loop runs
+- THEN it opens a PR writing the baseline file and persists the new baseline