Vague EVAL.ts conditions

[damoonsh]

Bug Report

Describe the bug

build-tests-001-rls-tenant-isolation grades a hidden process requirement that is not stated in the user-facing prompt.

PROMPT.md only asks the agent to write database tests, run them, and share results. It does not mention supabase/tests/, pgTAP, or supabase test db.

EVAL.ts fails the rollout unless:

1. *.sql files exist under supabase/tests/ (via find supabase/tests -name '*.sql'), and
2. the scorer’s own supabase test db run reports Tests > 0 with at least one passing test, and
3. a transcript judge correctly names the posts isolation bug (a policy fix is not required).

Agents that correctly find the posts leak and run real SQL checks via psql / ad-hoc files elsewhere get reward 0 (often check score 1/3), because gates 1–2 ignore those tests. That creates false negatives: right audit science, wrong filesystem/CLI convention.

Example from Muse-Glimmer-30B runs on this task: medium (±DFlash) failed all repetitions with a correct diagnosis but tests outside supabase/tests/; low (±DFlash) passed when files were placed under supabase/tests/ and run with supabase test db. Medium often used more reasoning tokens than a successful low pass and still failed.

To Reproduce

1. Open the eval at evals/build-tests-001-rls-tenant-isolation/.
2. Read the agent prompt in PROMPT.md — only this body text is shown to the model:

Can you audit the tenant isolation on our tables? Write some database tests covering both the happy path and the negative case, run them, and share the results.

3. Compare to the scorer in EVAL.ts. All three checks must pass:

const checks: CheckResult[] = [
  await checkTestFilesExist(ctx),
  ...(await checkTestResults(ctx)),
  await checkAgentDiagnosis(ctx),
];
return { passed: checks.every((c) => c.passed), checks };

4. Gate A — path only (any tests outside this tree fail):

async function checkTestFilesExist(
  ctx: LocalStackEvalContext
): Promise<CheckResult> {
  const name = 'pgTAP test file(s) written under supabase/tests/';
  const result = await ctx.exec(
    "find supabase/tests -name '*.sql' 2>/dev/null"
  );
  const files = result.stdout.trim().split('\n').filter(Boolean);
  return {
    name,
    passed: files.length > 0,
    notes:
      files.length > 0
        ? `${files.length} file(s): ${files.join(', ')}`
        : 'no .sql files found under supabase/tests/',
  };
}

5. Gate B — scorer re-runs supabase test db only; it does not credit the agent’s own psql runs from the transcript:

async function checkTestResults(
  ctx: LocalStackEvalContext
): Promise<CheckResult[]> {
  const result = await ctx.exec('supabase test db 2>&1', {
    timeoutMs: 120_000,
  });
  // parses Tests=/Failed: from pg_prove output
  return [
    {
      name: 'pgTAP isolation tests ran and pass',
      passed: ranTests && passedTests > 0,
      notes: summary,
    },
  ];
}

6. Gate C — diagnosis judge (excerpt): the agent does not need to supply the exact SQL fix, but must identify posts (not notes) and treat pgTAP results as authoritative.

7. Reproduce with an agent that:

   • correctly identifies the broken posts SELECT policy, and
   • writes/runs isolation SQL via psql (e.g. at repo root or /tmp), without creating supabase/tests/**/*.sql.

8. Observe score notes such as no .sql files found under supabase/tests/ and Files=0, Tests=0 / NOTESTS → reward 0, even when the diagnosis check passes.

Expected behavior

One of:

1. Prompt matches scorer: PROMPT.md explicitly requires pgTAP under supabase/tests/ and running supabase test db (and still does not require a policy fix), or
2. Scorer matches prompt: EVAL.ts credits equivalent isolation testing visible in the workspace and/or transcript (not only find supabase/tests + scorer-owned supabase test db), with clear partial/full credit rules, or
3. Hybrid: keep the preferred Supabase path in the prompt, and add partial credit when the agent clearly ran valid isolation tests another way and diagnosed posts correctly.

An agent that does the scientific work the prompt asks for should not look identical on the leaderboard to an agent that never tested, solely because files were not under supabase/tests/.

System information

• Eval: evals/build-tests-001-rls-tenant-isolation/
• Files involved: PROMPT.md, EVAL.ts, README.md (design notes mention pgTAP / supabase test db but are not the agent prompt)
• Scorer: LocalStackScorer from @supabase-evals/core
• Evidence models: Meta Muse-Glimmer-30B {low, low:DFlash, medium, medium:DFlash}

Additional context

• Win condition is not “fix the policy SQL.” Gate C’s rubric says the agent does not need the exact SQL fix; intended path is discoverable pgTAP + diagnosis from results.
• README.md design notes assume supabase test db as the authoritative signal versus misleading migration comments. That product intent is fine; the bug is enforcing it without stating it in PROMPT.md.
• Suggested acceptance criteria:
  • PROMPT states every hard path/CLI requirement EVAL enforces, or EVAL documents an alternate credit path
  • Running isolation tests via psql outside supabase/tests/ is either clearly disallowed in the prompt or can earn partial/full credit
  • Existing solutions that already use supabase/tests/ + supabase test db still pass