diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 00000000..f1136718 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,111 @@ +# AGENTS.md — supabase/evals + +Instructions for coding agents working in this repo. Humans: start with +[README.md](README.md) and [CONTRIBUTING.md](CONTRIBUTING.md). + +## What this repo is + +Evals for Supabase AI agents. An eval run is `agent + inputs -> score`. The +three inputs a change usually targets: the **skills** tree (in this repo), +the **MCP server** (external checkout), and **docs** content (external +supabase/supabase checkout). + +## Verifying a change against the evals (`pnpm eval`) + +Use `pnpm eval` for local and scheduled runs. It reads the current skills tree +and writes the result with its provenance receipt to +`results//.json`. + +```bash +pnpm eval -- --strict --eval --experiment --runs 3 +pnpm eval -- list +pnpm docs:local --docs +``` + +Use `--strict` for verification. It makes missing credentials, skills, local +stack support, and other error-class skips fail with exit 1. Intentional +`--skip-existing` and eval-filter skips stay nonfatal. + +Per input: + +- **Skill edited** (in `skills/`): no sync step. Run `pnpm eval -- --strict + --eval `. +- **MCP server edited** (external checkout): build that checkout, then add + `--mcp `. The runner validates the built server before spend. +- **Docs page edited** (external supabase/supabase checkout): use + `pnpm docs:local up`, `seed`, and `api` for the local index lifecycle. Seeding + costs about $0.12 OpenAI and asks before it starts. Keep `api` running, then + use `pnpm eval -- --strict --eval --content-api + http://127.0.0.1:3001/docs/api/graphql --mcp `. + `--content-api` needs `--mcp` while the harness pin is below v0.10.0. The + runner refuses a build without the `--content-api-url` flag before spend. + The docs checkout must include the lint-warnings authentication fix from + supabase/supabase#48364 or provide its required GitHub credentials. + +Score docs evals on retrieval (`docs.calls` and content returned by +`search_docs`). Tools-mode agents can also fetch production pages, which can +hide a local retrieval failure. + +Receipts include the host state, MCP override state, and content API URL in the +canonical result under `results/`. + +## Interpreting results — rules, not suggestions + +- **The published results in this repo are not a control arm.** They came from + the scheduled CI world (published MCP package, prod docs index, model state + at refresh time), so a difference against them is a signal consistent with + your edit and never proof the edit caused it. For a real two-arm comparison, + run the eval yourself before and after the change with everything else held + constant. +- **Single runs are noisy.** Before claiming improvement or regression, run + `--runs 3` and read check-level results, not just pass/fail. +- **MCP changes: judge by tool-call activation.** An eval can pass without + ever calling the tool you changed. Confirm the changed tool was actually + exercised (the result JSON records tool calls) before concluding anything. +- **Docs changes: the eval must be able to see the docs.** Use a tools-mode + (`interface: mcp`) eval whose answer lives in the edited page and is + reached via `search_docs`. CLI-scaffold evals can pass regardless of docs. +- **Custom evals work the same way.** `pnpm eval -- --strict --eval ` does + not require a published result. + +## Validating a dependency PR (e.g. supabase/mcp) + +1. **Baseline-proof first**: build the dependency's MAIN and run the chosen + eval(s) against it before the PR build — version pins hide fixture drift + (platform-lite tracks the pinned `MCP_SERVER_VERSION`, not your local + build's line; the runner warns on version mismatch). +2. Fixture or eval support living in an unmerged evals PR? Apply it into the + worktree as plain working-tree state: `gh pr diff | git apply`. + Receipts record the dirty tree, so runs stay attributable. +3. Run the PR build with `--mcp `; a main-FAIL -> PR-PASS flip with + everything else constant is a true two-arm comparison on the dependency + axis, which is why step 1 is worth the spend. +4. **Judge by tool-call activation, not pass/fail**: confirm the changed tool + was called, and unwrap `` envelopes in `toolCalls[]` + before reading results — errors hide inside them. Note that claude-code + records endpoints with an `mcp____` prefix; match with + `.endsWith('')`. + +## Spend rules + +- Eval runs cost model tokens. `pnpm docs:local seed` costs about $0.12 OpenAI + per invocation. State the cost and get user confirmation before unrequested + paid steps. +- Strict mode refuses pre-spend errors. Do not work around these gates. +- Zero-cost checks: `pnpm --filter @supabase-evals/framework test:local`, + `pnpm eval -- list`, and `pnpm eval:dry`. + +## Conventions + +- Keys live in `.env` at the repo root: `ANTHROPIC_API_KEY`, plus + `OPENAI_API_KEY` for the docs loop AND for any eval whose scorer uses the + LLM judge (an OpenAI grader model runs even when the agent under test is + Claude). Never hardcode or echo key values. +- Model/agent selection = experiment id. To test an unlisted model, add a + small `experiments/.ts` (copy an existing file's shape) rather than + editing a published experiment in place. +- `results/` and `.local-docs/` are outputs. Never commit their contents. +- Verify with `pnpm check` (typecheck + core/sandbox tests) and + `pnpm format:check` (biome) before pushing. +- New evals: follow [CONTRIBUTING.md](CONTRIBUTING.md) (suite choice, + `motivation:` frontmatter, scorer shape). diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 00000000..5514a3e2 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1 @@ +See [AGENTS.md](AGENTS.md) for agent instructions in this repo. diff --git a/README.md b/README.md index 31527982..42d73151 100644 --- a/README.md +++ b/README.md @@ -78,6 +78,56 @@ Start the web app development server: pnpm web ``` +## Local development loop (`pnpm eval`) + +Use the same entrypoint for local verification and scheduled runs: + +```bash +pnpm eval -- --strict --eval --experiment --runs 3 +pnpm eval -- list +``` + +`--strict` changes error-class skips into exit 1 failures. It catches missing +credentials, skills, local stack support, and invalid overrides before spend. +Every completed run writes its result and provenance receipt to +`results//.json`. + +- **Skills**: edit the skills tree in this repo. The harness reads it as-is. +- **MCP**: build a local + [`mcp-server-supabase`](https://github.com/supabase/mcp) checkout, then add + `--mcp `. The runner validates the built entrypoint first. +- **Docs**: serve a local docs content API from a supabase/supabase checkout: + + ```bash + pnpm docs:local up --docs + pnpm docs:local seed + pnpm docs:local api + pnpm eval -- --strict --eval --content-api http://127.0.0.1:3001/docs/api/graphql --mcp + pnpm docs:local down + ``` + + Keep `docs:local api` running during the eval. Seeding uses the docs app + pipeline, costs about $0.12 OpenAI, and asks before it starts. + + `--content-api` needs a local MCP build while the harness pin is below + v0.10.0. The runner refuses an older build without the + `--content-api-url` flag before spend. This requirement ends when the pin + reaches v0.10.0. + + The docs checkout must include + [supabase/supabase#48364](https://github.com/supabase/supabase/pull/48364) or + provide the credentials required by the lint-warnings source. Without that + fix, the seed stops before embedding. + +Keys go in `.env` at the repo root. Agent providers use their matching keys. +Docs seeding and judge-scored evals use `OPENAI_API_KEY`. + +Run the zero-cost controller self-test with: + +```bash +pnpm --filter @supabase-evals/framework test:local +``` + ## Eval Shape Every eval contains: diff --git a/apps/framework/scripts/docs/sentry-stub-register.mjs b/apps/framework/scripts/docs/sentry-stub-register.mjs index 0c425d1b..23bf93fa 100644 --- a/apps/framework/scripts/docs/sentry-stub-register.mjs +++ b/apps/framework/scripts/docs/sentry-stub-register.mjs @@ -1,6 +1,6 @@ // fallow-ignore-file unused-file -- loaded at runtime (spawned/injected by local-docs.ts), never statically imported // Registers a resolve hook that short-circuits '@sentry/nextjs' to the local -// no-op stub. Injected via NODE_OPTIONS from `pnpm local docs api`; chains with tsx's +// no-op stub. Injected via NODE_OPTIONS from `pnpm docs:local api`; chains with tsx's // own hooks (ours only intercepts the one specifier). Uses module.register() // (Node 20.6+) rather than registerHooks() (22.15+) — mise pins node "22", // which an older 22.x install satisfies. diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts index 06d884d5..fe766854 100644 --- a/packages/core/src/index.ts +++ b/packages/core/src/index.ts @@ -978,8 +978,8 @@ export function supabaseMcpServer( // `rewriteLoopback` then maps 127.0.0.1 -> host.docker.internal so the // host-side API is actually reachable from in there. Flag support landed // in supabase/mcp#343 and shipped in v0.10.0, so it works on the npx path - // too once MCP_SERVER_VERSION reaches that; below it, `pnpm local` refuses - // --content-api unless --mcp supplies a build that has the flag. + // too once MCP_SERVER_VERSION reaches that; below it, `pnpm eval` + // refuses --content-api unless --mcp supplies a build that has the flag. const contentApiUrl = process.env.SUPABASE_CONTENT_API_URL; if (contentApiUrl) serverArgs.push('--content-api-url', contentApiUrl);