From ad53e4feeb7d321cd3010db3b513fd910741372b Mon Sep 17 00:00:00 2001 From: Barry Roodt Date: Thu, 13 Aug 2026 17:25:08 +0200 Subject: [PATCH 1/2] docs: the local development loop README gets a "Local development loop" section covering the three agent inputs and how to override each: edit the skills tree and just run, build the mcp repo anywhere and pass --mcp, serve docs from your own supabase/supabase checkout and pass --content-api. It states where the keys go, and that `compare` is a screen rather than causal proof. It also records the one live limitation: `docs seed` needs a docs checkout carrying supabase/supabase#48364, which is still open. Without it fetchAllSources unconditionally awaits the lint-warnings source, whose loader needs the docs GitHub App, and a shared Promise.all turns that into a full abort before any embedding, so a failed attempt costs nothing. Two upstream rough edges are named rather than glossed: the seed exits 0 while silently failing 22 reference pages whose sections exceed the 8192-token embedding limit, and a local index has no partner-integration pages because that source reads the hosted misc project. AGENTS.md is the agent-facing version: which loop to run for a given kind of change, the spend rules for anything that costs money, and the same limitations stated as operational guidance. CLAUDE.md points at it. Verified: smoke 27/27, tsc and biome clean. --- AGENTS.md | 113 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ CLAUDE.md | 1 + README.md | 57 +++++++++++++++++++++++++++ 3 files changed, 171 insertions(+) create mode 100644 AGENTS.md create mode 100644 CLAUDE.md diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 00000000..df03b6e9 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,113 @@ +# AGENTS.md — supabase/evals + +Instructions for coding agents working in this repo. Humans: start with +[README.md](README.md) and [CONTRIBUTING.md](CONTRIBUTING.md). + +## What this repo is + +Evals for Supabase AI agents. An eval run is `agent + inputs -> score`. The +three inputs a change usually targets: the **skills** tree (in this repo), +the **MCP server** (external checkout), and **docs** content (external +supabase/supabase checkout). + +## Verifying a change against the evals (`pnpm local`) + +Use the local runner for all "did my change help / did it regress?" work. +It never mutates git state, so it is safe alongside in-flight work. + +```bash +pnpm local run [--experiment ] [--runs N] [--mcp ] [--content-api ] +pnpm local experiments # experiments + which have published results +pnpm local docs --docs +``` + +Per input: + +- **Skill edited** (in `skills/`): no sync step — `pnpm local run `. +- **MCP server edited** (external checkout): `pnpm build` in that checkout, + then `pnpm local run --mcp `. +- **Docs page edited** (external supabase/supabase checkout): + `pnpm local docs seed --yes` to re-embed (**~$0.12 OpenAI — see spend + rules**), keep `pnpm local docs api` running in a separate terminal, then + `pnpm local run --content-api http://127.0.0.1:3001/docs/api/graphql --mcp `. + `--content-api` needs `--mcp` while the harness pin is below v0.10.0: the + `--content-api-url` flag it forwards landed in supabase/mcp#343 and shipped in + v0.10.0, so the pinned package ignores it and `search_docs` would query + production docs while the receipt claimed otherwise. The runner refuses + pre-spend, and stops asking for `--mcp` once the pin reaches v0.10.0. + **`docs seed` currently fails against a vanilla docs checkout**: the pipeline + unconditionally loads its lint-warnings source, which needs a GitHub App + (`DOCS_GITHUB_APP_*`, no token fallback), and one `Promise.all` makes that + fatal. It aborts before embedding, so a retry costs nothing but achieves + nothing — don't loop on it. The leg needs an index seeded another way until a + skip flag lands upstream. + Score docs evals on retrieval (`docs.calls`, canary content coming back out of + `search_docs`), not on the answer text: tools mode also exposes + `WebSearch`/`WebFetch`, and an edit that contradicts the live page invites the + agent to fetch production and reject the local content as injection (observed). + +Receipts land in `results-local/` (git-ignored): treatment provenance, meaning +the host SHA + dirty state and the override paths with their git state. + +## Interpreting results — rules, not suggestions + +- **The published results in this repo are not a control arm.** They came from + the scheduled CI world (published MCP package, prod docs index, model state + at refresh time), so a difference against them is a signal consistent with + your edit and never proof the edit caused it. For a real two-arm comparison, + run the eval yourself before and after the change with everything else held + constant. +- **Single runs are noisy.** Before claiming improvement or regression, run + `--runs 3` and read check-level results, not just pass/fail. +- **MCP changes: judge by tool-call activation.** An eval can pass without + ever calling the tool you changed. Confirm the changed tool was actually + exercised (the result JSON records tool calls) before concluding anything. +- **Docs changes: the eval must be able to see the docs.** Use a tools-mode + (`interface: mcp`) eval whose answer lives in the edited page and is + reached via `search_docs`. CLI-scaffold evals can pass regardless of docs. +- **Custom evals work the same way.** `pnpm local run` does not need the eval + to appear in any published export. + +## Validating a dependency PR (e.g. supabase/mcp) + +1. **Baseline-proof first**: build the dependency's MAIN and run the chosen + eval(s) against it before the PR build — version pins hide fixture drift + (platform-lite tracks the pinned `MCP_SERVER_VERSION`, not your local + build's line; the runner warns on version mismatch). +2. Fixture or eval support living in an unmerged evals PR? Apply it into the + worktree as plain working-tree state: `gh pr diff | git apply`. + Receipts record the dirty tree, so runs stay attributable. +3. Run the PR build with `--mcp `; a main-FAIL -> PR-PASS flip with + everything else constant is a true two-arm comparison on the dependency + axis, which is why step 1 is worth the spend. +4. **Judge by tool-call activation, not pass/fail**: confirm the changed tool + was called, and unwrap `` envelopes in `toolCalls[]` + before reading results — errors hide inside them. Note that claude-code + records endpoints with an `mcp____` prefix; match with + `.endsWith('')`. + +## Spend rules + +- Eval runs cost model tokens; `pnpm local docs seed` costs **~$0.12 OpenAI + per invocation**. State the cost and get user confirmation before + running paid steps the user did not explicitly request. +- The runner refuses pre-spend on invalid eval metadata, unknown + experiments, and bad `--mcp` paths — do not work around these gates. +- Zero-cost checks: `pnpm --filter @supabase-evals/framework test:local` + (runner self-test), `pnpm local experiments`, `pnpm eval:dry`. + +## Conventions + +- Keys live in `.env` at the repo root: `ANTHROPIC_API_KEY`, plus + `OPENAI_API_KEY` for the docs loop AND for any eval whose scorer uses the + LLM judge (an OpenAI grader model runs even when the agent under test is + Claude). Never hardcode or echo key values. +- Model/agent selection = experiment id. To test an unlisted model, add a + small `experiments/.ts` (copy an existing file's shape) rather than + editing a published experiment in place. +- `results/`, `results-local/`, and `.local-docs/` are outputs — never + commit their contents. +- Verify with `pnpm check` (typecheck + core/sandbox tests) and + `pnpm format:check` (biome) before pushing. +- New evals: follow [CONTRIBUTING.md](CONTRIBUTING.md) (suite choice, + `motivation:` frontmatter, scorer shape). diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 00000000..5514a3e2 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1 @@ +See [AGENTS.md](AGENTS.md) for agent instructions in this repo. diff --git a/README.md b/README.md index 31527982..b882b05e 100644 --- a/README.md +++ b/README.md @@ -78,6 +78,63 @@ Start the web app development server: pnpm web ``` +## Local development loop (`pnpm local`) + +Testing a change to an agent input — a skill, a local build of +[`mcp-server-supabase`](https://github.com/supabase/mcp), or an edited docs +page — against the evals, without touching git state: + +```bash +pnpm local run [--experiment ] [--mcp ] [--content-api ] +pnpm local experiments # list experiments + which have published results +``` + +- **Skills**: edit the skills tree in this repo and just `run` — the harness + reads it as-is. +- **MCP**: clone + build the mcp repo anywhere, then `--mcp ` + (sets `SUPABASE_MCP_SERVER_PATH`, so `search_docs` and friends run your build). +- **Docs**: serve a local docs content API from your own supabase/supabase + checkout, then point runs at it: + + ```bash + pnpm local docs up --docs + pnpm local docs seed # full embed via the docs app's pipeline (~$0.12 OpenAI; asks first) + pnpm local docs api # keep running in a separate terminal + # --content-api needs a local mcp build while the harness pin is below v0.10.0: + # the --content-api-url flag landed in supabase/mcp#343 and shipped in v0.10.0, + # so the pinned server ignores it and search_docs would silently hit production + # docs. Refused pre-spend; --mcp stops being required once the pin catches up. + pnpm local run --content-api http://127.0.0.1:3001/docs/api/graphql --mcp + ``` + + **Known limitation — `docs seed` needs a docs checkout containing + [supabase/supabase#48364](https://github.com/supabase/supabase/pull/48364).** + Without it, `fetchAllSources()` unconditionally awaits the lint warnings source, + whose loader requires the docs GitHub App, and one shared `Promise.all` turns + that into a full abort before any embedding (so it costs nothing). That PR adds + a token rung below the App, `GH_TOKEN` then `GITHUB_TOKEN`, which is all a + contributor needs: `export GH_TOKEN=$(gh auth token)`. Until it merges, check + that branch out in the checkout you pass to `--docs`. + + Verified end to end against a checkout carrying it, with the `NEXT_PUBLIC_MISC_*` + wiring `docs seed` supplies: the seed completes (1901 sources, 7890 sections) and + a tools-mode eval's `search_docs` returns content that exists only in the local + index. Two rough edges to expect, both upstream: the seed exits 0 while silently + failing 22 `/reference/{javascript,dart}` pages whose sections exceed the + embedding model's 8192-token limit, and a local index has no partner-integration + pages, since that source reads the hosted misc project. Neither blocked the + tested guide-page eval, but an eval whose answer lives in those reference pages + would find them missing from the index. + +Every run writes a provenance receipt to `results-local/` (host SHA + dirty +state, override paths and their git state), so the world a result came from is +recorded in the artifact rather than in your memory. + +Keys go in `.env` at the repo root: `ANTHROPIC_API_KEY`, and `OPENAI_API_KEY` +for the docs loop and for judge-scored evals (the LLM judge is an OpenAI +grader model, regardless of the agent under test). Zero-cost self-test: `pnpm --filter +@supabase-evals/framework test:local`. + ## Eval Shape Every eval contains: From 7c703061b3e79747e0f2ece9b0a222a0ee5c3bf0 Mon Sep 17 00:00:00 2001 From: Barry Roodt Date: Tue, 18 Aug 2026 12:13:30 +0200 Subject: [PATCH 2/2] docs: use the unified eval CLI --- AGENTS.md | 82 +++++++++-------- README.md | 87 +++++++++---------- .../scripts/docs/sentry-stub-register.mjs | 2 +- packages/core/src/index.ts | 4 +- 4 files changed, 83 insertions(+), 92 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index df03b6e9..f1136718 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -10,44 +10,44 @@ three inputs a change usually targets: the **skills** tree (in this repo), the **MCP server** (external checkout), and **docs** content (external supabase/supabase checkout). -## Verifying a change against the evals (`pnpm local`) +## Verifying a change against the evals (`pnpm eval`) -Use the local runner for all "did my change help / did it regress?" work. -It never mutates git state, so it is safe alongside in-flight work. +Use `pnpm eval` for local and scheduled runs. It reads the current skills tree +and writes the result with its provenance receipt to +`results//.json`. ```bash -pnpm local run [--experiment ] [--runs N] [--mcp ] [--content-api ] -pnpm local experiments # experiments + which have published results -pnpm local docs --docs +pnpm eval -- --strict --eval --experiment --runs 3 +pnpm eval -- list +pnpm docs:local --docs ``` +Use `--strict` for verification. It makes missing credentials, skills, local +stack support, and other error-class skips fail with exit 1. Intentional +`--skip-existing` and eval-filter skips stay nonfatal. + Per input: -- **Skill edited** (in `skills/`): no sync step — `pnpm local run `. -- **MCP server edited** (external checkout): `pnpm build` in that checkout, - then `pnpm local run --mcp `. -- **Docs page edited** (external supabase/supabase checkout): - `pnpm local docs seed --yes` to re-embed (**~$0.12 OpenAI — see spend - rules**), keep `pnpm local docs api` running in a separate terminal, then - `pnpm local run --content-api http://127.0.0.1:3001/docs/api/graphql --mcp `. - `--content-api` needs `--mcp` while the harness pin is below v0.10.0: the - `--content-api-url` flag it forwards landed in supabase/mcp#343 and shipped in - v0.10.0, so the pinned package ignores it and `search_docs` would query - production docs while the receipt claimed otherwise. The runner refuses - pre-spend, and stops asking for `--mcp` once the pin reaches v0.10.0. - **`docs seed` currently fails against a vanilla docs checkout**: the pipeline - unconditionally loads its lint-warnings source, which needs a GitHub App - (`DOCS_GITHUB_APP_*`, no token fallback), and one `Promise.all` makes that - fatal. It aborts before embedding, so a retry costs nothing but achieves - nothing — don't loop on it. The leg needs an index seeded another way until a - skip flag lands upstream. - Score docs evals on retrieval (`docs.calls`, canary content coming back out of - `search_docs`), not on the answer text: tools mode also exposes - `WebSearch`/`WebFetch`, and an edit that contradicts the live page invites the - agent to fetch production and reject the local content as injection (observed). - -Receipts land in `results-local/` (git-ignored): treatment provenance, meaning -the host SHA + dirty state and the override paths with their git state. +- **Skill edited** (in `skills/`): no sync step. Run `pnpm eval -- --strict + --eval `. +- **MCP server edited** (external checkout): build that checkout, then add + `--mcp `. The runner validates the built server before spend. +- **Docs page edited** (external supabase/supabase checkout): use + `pnpm docs:local up`, `seed`, and `api` for the local index lifecycle. Seeding + costs about $0.12 OpenAI and asks before it starts. Keep `api` running, then + use `pnpm eval -- --strict --eval --content-api + http://127.0.0.1:3001/docs/api/graphql --mcp `. + `--content-api` needs `--mcp` while the harness pin is below v0.10.0. The + runner refuses a build without the `--content-api-url` flag before spend. + The docs checkout must include the lint-warnings authentication fix from + supabase/supabase#48364 or provide its required GitHub credentials. + +Score docs evals on retrieval (`docs.calls` and content returned by +`search_docs`). Tools-mode agents can also fetch production pages, which can +hide a local retrieval failure. + +Receipts include the host state, MCP override state, and content API URL in the +canonical result under `results/`. ## Interpreting results — rules, not suggestions @@ -65,8 +65,8 @@ the host SHA + dirty state and the override paths with their git state. - **Docs changes: the eval must be able to see the docs.** Use a tools-mode (`interface: mcp`) eval whose answer lives in the edited page and is reached via `search_docs`. CLI-scaffold evals can pass regardless of docs. -- **Custom evals work the same way.** `pnpm local run` does not need the eval - to appear in any published export. +- **Custom evals work the same way.** `pnpm eval -- --strict --eval ` does + not require a published result. ## Validating a dependency PR (e.g. supabase/mcp) @@ -88,13 +88,12 @@ the host SHA + dirty state and the override paths with their git state. ## Spend rules -- Eval runs cost model tokens; `pnpm local docs seed` costs **~$0.12 OpenAI - per invocation**. State the cost and get user confirmation before - running paid steps the user did not explicitly request. -- The runner refuses pre-spend on invalid eval metadata, unknown - experiments, and bad `--mcp` paths — do not work around these gates. -- Zero-cost checks: `pnpm --filter @supabase-evals/framework test:local` - (runner self-test), `pnpm local experiments`, `pnpm eval:dry`. +- Eval runs cost model tokens. `pnpm docs:local seed` costs about $0.12 OpenAI + per invocation. State the cost and get user confirmation before unrequested + paid steps. +- Strict mode refuses pre-spend errors. Do not work around these gates. +- Zero-cost checks: `pnpm --filter @supabase-evals/framework test:local`, + `pnpm eval -- list`, and `pnpm eval:dry`. ## Conventions @@ -105,8 +104,7 @@ the host SHA + dirty state and the override paths with their git state. - Model/agent selection = experiment id. To test an unlisted model, add a small `experiments/.ts` (copy an existing file's shape) rather than editing a published experiment in place. -- `results/`, `results-local/`, and `.local-docs/` are outputs — never - commit their contents. +- `results/` and `.local-docs/` are outputs. Never commit their contents. - Verify with `pnpm check` (typecheck + core/sandbox tests) and `pnpm format:check` (biome) before pushing. - New evals: follow [CONTRIBUTING.md](CONTRIBUTING.md) (suite choice, diff --git a/README.md b/README.md index b882b05e..42d73151 100644 --- a/README.md +++ b/README.md @@ -78,62 +78,55 @@ Start the web app development server: pnpm web ``` -## Local development loop (`pnpm local`) +## Local development loop (`pnpm eval`) -Testing a change to an agent input — a skill, a local build of -[`mcp-server-supabase`](https://github.com/supabase/mcp), or an edited docs -page — against the evals, without touching git state: +Use the same entrypoint for local verification and scheduled runs: ```bash -pnpm local run [--experiment ] [--mcp ] [--content-api ] -pnpm local experiments # list experiments + which have published results +pnpm eval -- --strict --eval --experiment --runs 3 +pnpm eval -- list ``` -- **Skills**: edit the skills tree in this repo and just `run` — the harness - reads it as-is. -- **MCP**: clone + build the mcp repo anywhere, then `--mcp ` - (sets `SUPABASE_MCP_SERVER_PATH`, so `search_docs` and friends run your build). -- **Docs**: serve a local docs content API from your own supabase/supabase - checkout, then point runs at it: +`--strict` changes error-class skips into exit 1 failures. It catches missing +credentials, skills, local stack support, and invalid overrides before spend. +Every completed run writes its result and provenance receipt to +`results//.json`. + +- **Skills**: edit the skills tree in this repo. The harness reads it as-is. +- **MCP**: build a local + [`mcp-server-supabase`](https://github.com/supabase/mcp) checkout, then add + `--mcp `. The runner validates the built entrypoint first. +- **Docs**: serve a local docs content API from a supabase/supabase checkout: ```bash - pnpm local docs up --docs - pnpm local docs seed # full embed via the docs app's pipeline (~$0.12 OpenAI; asks first) - pnpm local docs api # keep running in a separate terminal - # --content-api needs a local mcp build while the harness pin is below v0.10.0: - # the --content-api-url flag landed in supabase/mcp#343 and shipped in v0.10.0, - # so the pinned server ignores it and search_docs would silently hit production - # docs. Refused pre-spend; --mcp stops being required once the pin catches up. - pnpm local run --content-api http://127.0.0.1:3001/docs/api/graphql --mcp + pnpm docs:local up --docs + pnpm docs:local seed + pnpm docs:local api + pnpm eval -- --strict --eval --content-api http://127.0.0.1:3001/docs/api/graphql --mcp + pnpm docs:local down ``` - **Known limitation — `docs seed` needs a docs checkout containing - [supabase/supabase#48364](https://github.com/supabase/supabase/pull/48364).** - Without it, `fetchAllSources()` unconditionally awaits the lint warnings source, - whose loader requires the docs GitHub App, and one shared `Promise.all` turns - that into a full abort before any embedding (so it costs nothing). That PR adds - a token rung below the App, `GH_TOKEN` then `GITHUB_TOKEN`, which is all a - contributor needs: `export GH_TOKEN=$(gh auth token)`. Until it merges, check - that branch out in the checkout you pass to `--docs`. - - Verified end to end against a checkout carrying it, with the `NEXT_PUBLIC_MISC_*` - wiring `docs seed` supplies: the seed completes (1901 sources, 7890 sections) and - a tools-mode eval's `search_docs` returns content that exists only in the local - index. Two rough edges to expect, both upstream: the seed exits 0 while silently - failing 22 `/reference/{javascript,dart}` pages whose sections exceed the - embedding model's 8192-token limit, and a local index has no partner-integration - pages, since that source reads the hosted misc project. Neither blocked the - tested guide-page eval, but an eval whose answer lives in those reference pages - would find them missing from the index. - -Every run writes a provenance receipt to `results-local/` (host SHA + dirty -state, override paths and their git state), so the world a result came from is -recorded in the artifact rather than in your memory. - -Keys go in `.env` at the repo root: `ANTHROPIC_API_KEY`, and `OPENAI_API_KEY` -for the docs loop and for judge-scored evals (the LLM judge is an OpenAI -grader model, regardless of the agent under test). Zero-cost self-test: `pnpm --filter -@supabase-evals/framework test:local`. + Keep `docs:local api` running during the eval. Seeding uses the docs app + pipeline, costs about $0.12 OpenAI, and asks before it starts. + + `--content-api` needs a local MCP build while the harness pin is below + v0.10.0. The runner refuses an older build without the + `--content-api-url` flag before spend. This requirement ends when the pin + reaches v0.10.0. + + The docs checkout must include + [supabase/supabase#48364](https://github.com/supabase/supabase/pull/48364) or + provide the credentials required by the lint-warnings source. Without that + fix, the seed stops before embedding. + +Keys go in `.env` at the repo root. Agent providers use their matching keys. +Docs seeding and judge-scored evals use `OPENAI_API_KEY`. + +Run the zero-cost controller self-test with: + +```bash +pnpm --filter @supabase-evals/framework test:local +``` ## Eval Shape diff --git a/apps/framework/scripts/docs/sentry-stub-register.mjs b/apps/framework/scripts/docs/sentry-stub-register.mjs index 0c425d1b..23bf93fa 100644 --- a/apps/framework/scripts/docs/sentry-stub-register.mjs +++ b/apps/framework/scripts/docs/sentry-stub-register.mjs @@ -1,6 +1,6 @@ // fallow-ignore-file unused-file -- loaded at runtime (spawned/injected by local-docs.ts), never statically imported // Registers a resolve hook that short-circuits '@sentry/nextjs' to the local -// no-op stub. Injected via NODE_OPTIONS from `pnpm local docs api`; chains with tsx's +// no-op stub. Injected via NODE_OPTIONS from `pnpm docs:local api`; chains with tsx's // own hooks (ours only intercepts the one specifier). Uses module.register() // (Node 20.6+) rather than registerHooks() (22.15+) — mise pins node "22", // which an older 22.x install satisfies. diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts index 06d884d5..fe766854 100644 --- a/packages/core/src/index.ts +++ b/packages/core/src/index.ts @@ -978,8 +978,8 @@ export function supabaseMcpServer( // `rewriteLoopback` then maps 127.0.0.1 -> host.docker.internal so the // host-side API is actually reachable from in there. Flag support landed // in supabase/mcp#343 and shipped in v0.10.0, so it works on the npx path - // too once MCP_SERVER_VERSION reaches that; below it, `pnpm local` refuses - // --content-api unless --mcp supplies a build that has the flag. + // too once MCP_SERVER_VERSION reaches that; below it, `pnpm eval` + // refuses --content-api unless --mcp supplies a build that has the flag. const contentApiUrl = process.env.SUPABASE_CONTENT_API_URL; if (contentApiUrl) serverArgs.push('--content-api-url', contentApiUrl);