fix(ai): gate live test credentials behind an explicit opt-in

[kevinjosethomas]

Linear: ENG-5347 — https://linear.app/primeintellect/issue/ENG-5347

Context

packages/ai/test/oauth.ts (resolveApiKey) read the developer's legacy credential store ~/.pi/agent/auth.json, refreshed expired OAuth entries against the real provider token endpoint, and wrote the refreshed tokens back. Thirteen test modules call it at module top level, so a plain vitest --run in packages/ai touched real credentials before any skipIf gate, with no opt-in. test.sh only moved ~/.prime/agent/auth.json aside, not the .pi file the helper used.

Root cause: the helper hard-coded the real store path and had no gate. Developer workflow only; the shipped runtime does not import this helper.

Changes

• packages/ai/test/oauth.ts: resolveApiKey returns undefined without touching the filesystem or network unless PI_LIVE_TESTS=1. With the opt-in it reads only the file named by PI_TEST_AUTH_FILE (same JSON shape as auth.json) and writes refreshed tokens back to that file only. Pointing PI_TEST_AUTH_FILE at ~/.prime/agent/auth.json or ~/.pi/agent/auth.json throws.
• packages/ai/test/oauth-test-helper.test.ts: unit test with HOME set to a temp dir, sentinel stores at both real paths, fetch stubbed and fs spied. Asserts no reads, no network, no writes without the opt-in (or with the opt-in but no test file); reads only the test file and writes only the test file with the opt-in; refuses the real stores.
• test.sh: moves both ~/.prime/agent/auth.json and ~/.pi/agent/auth.json aside (and restores both), and unsets PI_LIVE_TESTS / PI_TEST_AUTH_FILE.
• Docs: "Running tests" section in packages/ai/README.md; notes in packages/coding-agent/docs/development.md and CONTRIBUTING.md.
• Changelog fragment packages/ai/.changes/eng-5347-test-credential-isolation.md.

No model-facing surface changes.

Validation

Local (worktree): npm run check clean. test/oauth-test-helper.test.ts 5/5 pass. anthropic-tool-name-normalization, anthropic-eager-tool-input-e2e, anthropic-oauth, openai-codex-oauth, github-copilot-oauth run: 6 passed, 32 skipped.

Prime sandbox (node:24-bookworm, user tester, fresh HOME per run, synthetic sentinel stores at both ~/.pi/agent/auth.json and ~/.prime/agent/auth.json, fetch stubbed, probe test recording behaviour):

• Before (main 427ea4c72, no env): resolveApiKey("anthropic") returned REFRESHED-ACCESS-5347, POSTed refresh_token=SENTINEL-LEGACY-PI-REFRESH-5347 to https://platform.claude.com/v1/oauth/token, rewrote ~/.pi/agent/auth.json with the refreshed tokens.
• After (branch, no env): returned undefined, zero outbound calls, both stores byte-identical.
• After (branch, PI_LIVE_TESTS=1 PI_TEST_AUTH_FILE=$HOME/test-auth.json): refresh POST carried only the test file's SENTINEL-TEST-FILE-REFRESH-5347; refreshed tokens written to the test file (0600); both real stores untouched.
• After (branch, PI_TEST_AUTH_FILE=$HOME/.pi/agent/auth.json): throws "must point at a dedicated test credential file", no reads, no network, no writes.
• Branch, no opt-in, sentinel stores present: all 13 resolveApiKey caller modules skip (3 passed / 311 skipped, the passes are non-credential cases); stores unchanged afterwards. Helper unit test + anthropic-oauth, openai-codex-oauth, github-copilot-oauth, mcp-oauth: 24/24 pass.
• test.sh with npm replaced by a reporting stub and PI_LIVE_TESTS=1 PI_TEST_AUTH_FILE=/x in the caller env: main leaves ~/.pi/agent/auth.json present and the opt-in set; branch moves both stores aside, unsets both vars, and restores both files afterwards.

Live provider suites were not run.

Note

Gate live test credentials behind PI_LIVE_TESTS=1 and PI_TEST_AUTH_FILE opt-in

• resolveApiKey in oauth.ts returns undefined without filesystem or network access unless live tests are explicitly enabled with a dedicated credential file
• The default test suite no longer reads, refreshes, or rewrites the real ~/.prime/agent/auth.json or ~/.pi/agent/auth.json stores; refreshed OAuth tokens persist to the configured test file with 0600 permissions
• test.sh now backs up and restores both real credential stores and unsets PI_LIVE_TESTS and PI_TEST_AUTH_FILE before running the suite
• New isolation tests in oauth-test-helper.test.ts cover disabled live tests, missing config, dedicated-file refresh, real-store rejection, and fixture preservation
• Risk: any developer relying on the old behavior of automatically using their real credential store during tests must now set both environment variables and provide a separate auth file

^{Macroscope summarized 1cc2247.}