Add Experiment Lifecycle Orchestrator (first long-horizon loop)

loops/experiment/README.md

@@ -0,0 +1,54 @@
+# experiment-orchestrator loop
+
+A **long-horizon, multi-step** loop that drives an experiment from hypothesis to
+decision — the flagship example of loopy's `longrun` primitives (durable state,
+human-approval gates, resumable plans).
+
+It is **not** a single-shot `Loop`; it advances a resumable **plan** via
+`runPlan`. Call `advanceExperiment(...)` repeatedly (e.g. on a schedule); each
+call resumes from the last pause and is a no-op once the plan completes.
+
+## The plan
+
+```
+design → approve-design (gate) → launch → bake (wait) → readout → decide (gate)
+```
+
+| Step | What happens | Pause? |
+|------|--------------|--------|
+| **design** | AI designer turns the hypothesis into an `ExperimentDesign` (variants, metric, guardrails, sample size, duration) | — |
+| **approve-design** | Human approval gate before spend | **blocked** until decided |
+| **launch** | Create the experiment on the platform; record launch time | — |
+| **bake** | Wait until results are available (long-horizon) | **waiting** until results land |
+| **readout** | Render a readout from results | — |
+| **decide** | Ship/kill/iterate — human gate; respects the recommendation | **blocked** until decided |
+
+A rejected design short-circuits the plan to `finalDecision: "rejected"` without
+launching.
+
+## Boundaries (injected, testable)
+
+- `services.designer` — the AI design step (`createExperimentDesigner` over the
+  OpenRouter provider).
+- `services.platform` — the experimentation platform (`launch` + `results`);
+  adapt to Statsig/Eppo/GrowthBook/LaunchDarkly.
+- A `StateStore` (memory or file) carries plan progress + experiment registry
+  across runs; gates are created from it.
+
+## Usage
+
+```ts
+import { advanceExperiment } from "loopy/loops/experiment";
+import { createFileStateStore } from "loopy";
+
+const store = createFileStateStore(".loopy/state");
+const result = await advanceExperiment("exp-cta", hypothesis, { designer, platform }, store);
+// result.status: "blocked" | "waiting" | "completed"; result.memory carries
+// design, experimentKey, results, readout, finalDecision as they are produced.
+```
+
+## Status
+
+Programmatic for now. A CLI verb to advance plans on a schedule
+(`loopy advance <plan>`) is a planned follow-up; it needs a persistent state
+location and a hypothesis registry.

loops/experiment/hooks/types.ts

@@ -0,0 +1,38 @@
+/** A testable hypothesis for an experiment. */
+export interface Hypothesis {
+  id: string;
+  statement: string;
+  /** the primary success metric */
+  metric: string;
+  /** metrics that must not regress */
+  guardrailMetrics?: string[];
+  /** the surface/area the experiment runs on */
+  surface?: string;
+}
+
+/** A concrete experiment design produced from a hypothesis. */
+export interface ExperimentDesign {
+  variants: string[];
+  metric: string;
+  guardrailMetrics: string[];
+  minSampleSize: number;
+  durationDays: number;
+}
+
+/** The outcome of a baked experiment. */
+export interface ExperimentResults {
+  significant: boolean;
+  metricDelta: number;
+  guardrailBreached: boolean;
+  recommendation: "ship" | "kill" | "iterate";
+}
+
+/** The AI step (driven by `playbook.md`): design an experiment from a hypothesis. */
+export type ExperimentDesigner = (hypothesis: Hypothesis) => Promise<ExperimentDesign>;
+
+/** The experimentation-platform boundary (Statsig/Eppo/GrowthBook/LaunchDarkly, etc.). */
+export interface ExperimentPlatform {
+  launch(hypothesis: Hypothesis, design: ExperimentDesign): Promise<{ experimentKey: string }>;
+  /** results once the experiment has baked, or null while still running */
+  results(experimentKey: string): Promise<ExperimentResults | null>;
+}

loops/experiment/index.ts

@@ -0,0 +1,168 @@
+import {
+  createGate,
+  runPlan,
+  type Gate,
+  type PlanRunResult,
+  type StateStore,
+  type Step,
+  type StepOutcome,
+} from "../../src/core/index.js";
+import type {
+  ExperimentDesign,
+  ExperimentDesigner,
+  ExperimentPlatform,
+  ExperimentResults,
+  Hypothesis,
+} from "./hooks/types.js";
+
+export type {
+  ExperimentDesign,
+  ExperimentDesigner,
+  ExperimentPlatform,
+  ExperimentResults,
+  Hypothesis,
+} from "./hooks/types.js";
+
+export interface ExperimentServices {
+  designer: ExperimentDesigner;
+  platform: ExperimentPlatform;
+}
+
+interface PlanInput {
+  planId: string;
+  hypothesis: Hypothesis;
+  services: ExperimentServices;
+  gate: Gate;
+}
+
+/** Typed read of a value previously merged into plan memory. */
+function mem<T>(memory: Readonly<Record<string, unknown>>, key: string): T | undefined {
+  return memory[key] as T | undefined;
+}
+
+const DAY_MS = 24 * 60 * 60 * 1000;
+
+/**
+ * The experiment lifecycle as a resumable plan:
+ *   design → approve-design (gate) → launch → bake (wait) → readout → decide (gate)
+ *
+ * Each step is safe to re-run; the plan persists progress via the StateStore and
+ * resumes from where it paused (a human gate or a long-horizon bake).
+ */
+export function experimentSteps(): Step<PlanInput>[] {
+  return [
+    {
+      name: "design",
+      async run({ input }): Promise<StepOutcome> {
+        const design = await input.services.designer(input.hypothesis);
+        return { status: "done", data: { design } };
+      },
+    },
+    {
+      name: "approve-design",
+      async run({ input, memory }): Promise<StepOutcome> {
+        const design = mem<ExperimentDesign>(memory, "design");
+        const gateId = `${input.planId}:design`;
+        const req = await input.gate.require(
+          gateId,
+          `Approve experiment design for "${input.hypothesis.statement}" ` +
+            `(${design?.durationDays ?? "?"}d, metric ${design?.metric ?? "?"}).`,
+        );
+        if (req.status === "pending") return { status: "blocked", gateId };
+        if (req.status === "rejected") {
+          return { status: "done", data: { rejected: true, finalDecision: "rejected" } };
+        }
+        return { status: "done" };
+      },
+    },
+    {
+      name: "launch",
+      async run({ input, memory, now }): Promise<StepOutcome> {
+        if (mem<boolean>(memory, "rejected")) return { status: "done" };
+        const design = mem<ExperimentDesign>(memory, "design");
+        if (!design) return { status: "done", data: { error: "missing design" } };
+        const { experimentKey } = await input.services.platform.launch(input.hypothesis, design);
+        return { status: "done", data: { experimentKey, launchedAtIso: now.toISOString() } };
+      },
+    },
+    {
+      name: "bake",
+      async run({ input, memory, now }): Promise<StepOutcome> {
+        if (mem<boolean>(memory, "rejected")) return { status: "done" };
+        const experimentKey = mem<string>(memory, "experimentKey");
+        if (!experimentKey) return { status: "done", data: { error: "missing experimentKey" } };
+
+        // The step owns the wait→done transition: complete as soon as results are
+        // available, otherwise keep waiting until the projected bake deadline.
+        const results = await input.services.platform.results(experimentKey);
+        if (results) return { status: "done", data: { results } };
+
+        const design = mem<ExperimentDesign>(memory, "design");
+        const launchedAtIso = mem<string>(memory, "launchedAtIso");
+        const launchedAt = launchedAtIso ? Date.parse(launchedAtIso) : now.getTime();
+        const untilIso = new Date(launchedAt + (design?.durationDays ?? 14) * DAY_MS).toISOString();
+        return { status: "waiting", untilIso };
+      },
+    },
+    {
+      name: "readout",
+      async run({ input, memory }): Promise<StepOutcome> {
+        if (mem<boolean>(memory, "rejected")) return { status: "done" };
+        const results = mem<ExperimentResults>(memory, "results");
+        return { status: "done", data: { readout: renderReadout(input.hypothesis, results) } };
+      },
+    },
+    {
+      name: "decide",
+      async run({ input, memory }): Promise<StepOutcome> {
+        if (mem<boolean>(memory, "rejected")) {
+          return { status: "done", data: { finalDecision: "rejected" } };
+        }
+        const results = mem<ExperimentResults>(memory, "results");
+        const gateId = `${input.planId}:decision`;
+        const req = await input.gate.require(
+          gateId,
+          `Ship decision for "${input.hypothesis.statement}": ` +
+            `recommended ${results?.recommendation ?? "?"} ` +
+            `(delta ${results?.metricDelta ?? "?"}, guardrail breached: ${results?.guardrailBreached ?? "?"}).`,
+        );
+        if (req.status === "pending") return { status: "blocked", gateId };
+        if (req.status === "rejected") return { status: "done", data: { finalDecision: "hold" } };
+        return { status: "done", data: { finalDecision: results?.recommendation ?? "iterate" } };
+      },
+    },
+  ];
+}
+
+/** Render a human-readable experiment readout. */
+export function renderReadout(hypothesis: Hypothesis, results?: ExperimentResults): string {
+  if (!results) return `## Experiment readout: ${hypothesis.statement}\n\n_No results available._`;
+  return [
+    `## Experiment readout: ${hypothesis.statement}`,
+    "",
+    `- Metric: \`${hypothesis.metric}\` — delta **${results.metricDelta}** ` +
+      `(${results.significant ? "significant" : "not significant"})`,
+    `- Guardrails breached: **${results.guardrailBreached ? "yes" : "no"}**`,
+    `- Recommendation: **${results.recommendation}**`,
+    "",
+    "_Generated by loopy experiment-orchestrator. Final ship/kill is human-gated._",
+  ].join("\n");
+}
+
+/**
+ * Advance one experiment's lifecycle by (re)running its plan. Call repeatedly
+ * (e.g. on a schedule); it resumes from the last gate/bake pause and is a no-op
+ * once completed. Returns the plan result, whose `memory` carries `design`,
+ * `experimentKey`, `results`, `readout`, and `finalDecision` as they are produced.
+ */
+export async function advanceExperiment(
+  planId: string,
+  hypothesis: Hypothesis,
+  services: ExperimentServices,
+  store: StateStore,
+  options?: { now?: () => Date },
+): Promise<PlanRunResult> {
+  const gate = createGate(store, options);
+  const input: PlanInput = { planId, hypothesis, services, gate };
+  return runPlan(planId, experimentSteps(), input, store, options);
+}

loops/experiment/playbook.md

@@ -0,0 +1,27 @@
+# Experiment Designer Playbook
+
+This playbook drives the **design** step of the experiment-orchestrator loop —
+the AI step that turns a hypothesis into a concrete A/B test design. The loop
+framework handles approval gates, launch, baking, readout, and the ship
+decision; this document tells the designer what to produce.
+
+The designer is supplied as `services.designer` (see `createExperimentDesigner`).
+It receives a `Hypothesis` and returns an `ExperimentDesign`.
+
+## Instructions
+
+1. **Pick variants.** Usually `["control", "<treatment>"]`; more only if justified.
+2. **Confirm the primary metric** from the hypothesis; restate it precisely.
+3. **Always include guardrail metrics** — the things that must not regress
+   (latency, error rate, revenue, churn proxy). Never ship without them.
+4. **Size the test realistically** — a `minSampleSize` and `durationDays` that
+   can plausibly detect the expected effect. Prefer a 1–2 week bake.
+5. **Return ONLY JSON**: `{ variants, metric, guardrailMetrics, minSampleSize, durationDays }`.
+
+## Guardrails (enforced by the loop, not optional)
+
+- Design is reviewed at a **human approval gate** before launch.
+- The final ship/kill is a **human gate**, never automatic — the loop only
+  recommends based on results.
+- Decisions are made on baked results (significance + guardrails), not on
+  intraday noise.

openspec/archive/add-experiment-orchestrator/proposal.md

@@ -0,0 +1,70 @@
+# Proposal: Experiment Lifecycle Orchestrator
+
+**Change ID:** `add-experiment-orchestrator`
+**Created:** 2026-06-22
+**Status:** Implementation Complete
+**Completed:** 2026-06-22
+
+---
+
+## Problem Statement
+
+The `longrun` primitives (durable state, approval gates, resumable plans) shipped
+with no loop consuming them. The Experiment Lifecycle Orchestrator — the
+top-differentiation product-level loop from the research — is the flagship that
+exercises all of them and proves the long-horizon model end-to-end.
+
+## Proposed Solution
+
+Add `loops/experiment/`: a resumable plan that drives an experiment
+`design → approve-design (gate) → launch → bake (wait) → readout → decide (gate)`
+via `runPlan`, persisting progress in a `StateStore` and pausing at human gates
+and the long-horizon bake. Boundaries are injected: an AI **designer**
+(`createExperimentDesigner` over the OpenRouter provider) and an
+**ExperimentPlatform** (`launch` + `results`). Advanced by calling
+`advanceExperiment(...)` repeatedly; resumes from the last pause; no-op once
+complete. A rejected design short-circuits without launching.
+
+## Scope
+
+### In Scope
+- `loops/experiment/` (types, plan/steps, `advanceExperiment`, readout, README, playbook)
+- `createExperimentDesigner` AI adapter
+- Full-lifecycle unit tests (gates, bake wait, completion, rejection) with fakes + memory store
+
+### Out of Scope
+- A CLI verb to advance plans on a schedule (`loopy advance`) — follow-up (needs a persistent state location + hypothesis registry)
+- Concrete platform connectors (Statsig/Eppo/GrowthBook) — injected boundary
+- Auto-ship (decision is always human-gated)
+
+## Impact Analysis
+
+| Component | Change | Details |
+|-----------|--------|---------|
+| Core | No | consumes `longrun` as-is |
+| Loops | Yes | new `loops/experiment/` (plan-based, not single-shot) |
+| AI provider | Yes | `createExperimentDesigner` |
+
+## Success Criteria
+
+- [x] Blocks at the design gate; resumes on approval.
+- [x] Waits during bake; completes the bake step once results land.
+- [x] Blocks at the decision gate; completes with the recommended decision on approval.
+- [x] Rejected design short-circuits to `finalDecision: "rejected"` without launching.
+- [x] State + progress persist and resume across separate `advanceExperiment` calls.
+
+## Risks & Mitigations
+
+| Risk | Prob | Impact | Mitigation |
+|------|------|--------|------------|
+| Acting on noisy/early results | Med | High | Bake completes only when platform reports results; decision is human-gated |
+| Runaway auto-ship | Low | High | No auto-ship — explicit decision gate |
+| Stuck plans | Low | Med | Steps own wait→done; resumable; rejection short-circuit |
+
+---
+
+## Archive Information
+
+**Archived:** 2026-06-22
+**Outcome:** Successfully implemented
+**Verification:** typecheck + lint + 111 tests + build all passing