Add 11 loops: model-upgrade-migration, PR-gate guardrails, stateful guardrails, long-horizon migrations#14
Conversation
Rank #7 from research deep-dive #3. A long-horizon loop on runPlan + Gate + StateStore that migrates to a new model version safely: - loops/model-upgrade-migration/: advanceModelUpgrade() runs a resumable plan baseline -> candidate -> diff -> approve (gate) -> apply. It evaluates a golden set on the current and candidate models (injected evaluate(modelId)), computes regressions + score delta (reusing prompt-eval's Scorecard/ regressions), blocks at a human approval gate, and on approval emits the model-id bump change set (result.bump); on rejection it holds. Programmatic like the experiment orchestrator (a PR adapter/CLI verb is a shared follow-up). Shipped via the OpenSpec cycle (openspec/specs/model-upgrade-migration.md). Validated: typecheck, lint, 174 tests (+2), clean build. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012CrMioorha3QtPUgWbtDbR
Wave 1 of the remaining roadmap loops (research deep-dive #3), built in parallel via subagents — deterministic, single-shot, report-PR output, with injected file-based boundaries; wired into the catalog + loopy run + package exports. - i18n-drift: missing/orphaned translation keys across locales - perf-budget: metric regressions vs a stored baseline - a11y-baseline: new accessibility violations vs a baseline - runbook-freshness: runbooks past their review interval Shipped via the OpenSpec cycle (4 specs in openspec/specs/). Validated: typecheck, lint, 200 tests (+26), clean build. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012CrMioorha3QtPUgWbtDbR
…ost-guardrail, eval-set-drift Four loops on the durable StateStore (+ Gate where human approval is warranted): - test-impact-budget: per-test runtime vs an EWMA baseline; flags regressions past a growth threshold, writes a report PR, rolls the baseline forward. - data-contract-guard: diffs the current schema vs an approved baseline; additive changes auto-record, breaking changes block behind a human Gate before a PR records the new baseline. - cost-guardrail: tracks per-resource idle streaks (advanced every run as a grace period) and proposes gated remediation once idle past minStreak; never deletes. - eval-set-drift: surfaces production categories missing from the eval set as proposed new eval cases, recording surfaced categories so each reports once. Each wires into the catalog, `loopy run` (env-file boundaries + file StateStore), and package exports, with unit tests over fakes and a memory StateStore. Specs + archives added; README/roadmap updated. Fixes a cost-guardrail accumulator bug: the idle streak is now advanced and persisted in detect (every run) instead of only in act, so it can actually reach the threshold; act reads the already-advanced value to avoid double-counting. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ration Two export-only, programmatic loops on the resumable `runPlan` primitive (the `model-upgrade-migration` pattern: an `advance*` function over a step plan, persisted under `plan:<planId>` in the StateStore, resumable across runs): - api-deprecation-rollout: drives an API deprecation across long-horizon stages — announce → grace-period (long-horizon wait) → verify-callers (drain) → approve-removal (human gate) → remove. It will not remove an API while callers remain, and removal is emitted only after a human approves the gate. - dep-major-migration: migrates one major dependency bump (the kind dep-updates deliberately skips) — verify the consumer build/tests against the candidate, human-gate a green result, then emit the manifest bump. A red build is never gated and never proposed; the failing log is surfaced instead. Each emits its change set as a FileChange[] for a caller/adapter to open as a PR. Package exports + specs + archives added; README/roadmap updated. With these, the complex-loops ranking and the stateful/long-horizon backlog are fully implemented. Validation: typecheck + lint + 249 tests + build all passing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
DCCA
left a comment
There was a problem hiding this comment.
Self-review
Reviewed the full diff (114 files, +5804 / −5, all four commits). Findings:
Consistency — all 11 loops follow the established contract and conventions: ESM NodeNext with .js relative imports, import type for type-only imports, *FromManifest factories with typeof config guards, deterministic renderers, and fail-safe detect/act. The wave-3 loops mirror model-upgrade-migration exactly (runPlan steps + an advance* export, resumable via plan:<planId>).
Boundaries — every external dependency is injected (timing/usage/schema/drift sources, verify, remainingCallers, readers) and faked in tests; no network in unit tests. CLI wiring reads env-file boundaries + a file StateStore, matching the wave-1 pattern.
Bug caught during integration — cost-guardrail persisted its idle streak only in act, but act only runs once the streak crosses the threshold, so the counter could never accumulate to fire. Fixed: the streak now advances+persists in detect (every run, which is what the grace period requires) and act reads the already-advanced value to avoid double-counting. Covered by the "builds a streak… then produces a PR once approved" and "never reaches the gate for a busy resource" tests.
Gates / safety — human-gated paths (data-contract-guard, cost-guardrail, api-deprecation-rollout, dep-major-migration) all hold until explicit approval and emit nothing destructive; cost-guardrail never auto-deletes, dep-major-migration never gates a red build, api-deprecation-rollout won't remove while callers remain.
Validation — typecheck, lint, 249 tests, and build all clean locally; CLI list/guidance smoke-tested. OpenSpec specs + archives present for all 11 loops.
LGTM — merging once the build check goes green.
Generated by Claude Code
Summary
Implements all remaining roadmap loops — the complex-loops ranking and the stateful / long-horizon backlog are now fully built out. 11 new loops across 4 commits, each following loopy's
trigger → detect → act → output → guardrailscontract with injected boundaries (fully unit-tested with fakes / a memoryStateStore).Model-upgrade migration (
415f3e9)model-upgrade-migration— golden-set diff → human-gated model switch onrunPlan(baseline → candidate → diff → approve → apply). Reusesprompt-eval-gate's scorecard/regression logic.Wave 1 — PR-gate & freshness guardrails (
5593467)Deterministic, single-shot, report-PR loops:
i18n-drift— flag missing/orphaned translation keys across localesperf-budget— flag performance regressions vs a stored baselinea11y-baseline— fail only on new accessibility violations vs a baselinerunbook-freshness— flag runbooks past their review intervalWave 2 — stateful guardrails (
619e69b)On the durable
StateStore(+ a humanGatewhere approval is warranted):test-impact-budget— flag tests grown past a rolling EWMA runtime baseline; rolls the baseline forwarddata-contract-guard— additive schema changes auto-record; breaking changes block behind a human gate before a PR records the new baselinecost-guardrail— track per-resource idle streaks (grace period); gated remediation report once idle pastminStreak— never auto-deleteseval-set-drift— surface production categories missing from the eval set; record surfaced categories so each reports onceWave 3 — long-horizon migrations (
954a0bb)Export-only programmatic
advance*loops on the resumablerunPlanprimitive (themodel-upgrade-migrationpattern; resumable viaplan:<planId>in theStateStore):api-deprecation-rollout— announce → grace-period (long-horizon wait) → caller-drain → human-gated removal. Won't remove an API while callers remain.dep-major-migration— migrate one major bump (the kinddep-updatesskips): verify the consumer build → human-gate a green result → emit the manifest bump. A red build is never gated or proposed; the failing log is surfaced.Each wave-2 loop is wired into the catalog,
loopy run(env-file boundaries + fileStateStore), and package exports; the wave-3 programmatic loops add package exports. Every loop has aloop.yaml, README, playbook, and tests.OpenSpec
Specs (source of truth) + archives added for all 11 loops under
openspec/specs/andopenspec/archive/;openspec/roadmap.mdandREADME.mdupdated.Validation
npm run typecheck/lint— cleannpm test— 249 tests passingnpm run build— cleanInventory: 26 loop directories, 21 runnable via the CLI catalog, 5 export-only programmatic long-horizon loops.
🤖 Generated with Claude Code
https://claude.ai/code/session_012CrMioorha3QtPUgWbtDbR
Generated by Claude Code