Skip to content

fix(evals): coalesce null output for Braintrust reporter and guard log() throws - #1530

Open
iroiro147 wants to merge 1 commit into
vercel:mainfrom
iroiro147:fix/braintrust-null-output-1405
Open

fix(evals): coalesce null output for Braintrust reporter and guard log() throws#1530
iroiro147 wants to merge 1 commit into
vercel:mainfrom
iroiro147:fix/braintrust-null-output-1405

Conversation

@iroiro147

Copy link
Copy Markdown
Contributor

What

A no-turn eval (e.g. t.target.dispatchSchedule(...) + t.check over DB state) legitimately produces result.output === null per the eval API's own derivation. Braintrust's validateAndSanitizeExperimentLogFullArgs rejects null/undefined output ("output must be specified"), and the throw escaped onEvalComplete, killing the entire eve eval run — remaining evals never executed and no artifacts were written, even though the crashing eval itself passed all its gates.

Two fixes in BraintrustReporter.onEvalComplete:

  1. output: result.result.output ?? "" — coalesce null to an empty string so the Braintrust SDK accepts it.
  2. try/catch around experiment.log() — any reporter throw is logged to stderr but does not abort the run. The remaining evals still execute and artifacts are written.

Regression tests

Added to braintrust.test.ts:

  • coalesces null output to empty string for no-turn evals — drives an eval result with output: null, asserts log receives output: "".
  • survives a log() throw without aborting the run — mocks log to throw "output must be specified", asserts no rethrow, then asserts a second call still reaches log (proving the next eval is not dropped).

pnpm exec vitest run --config vitest.unit.config.ts src/evals/runner/reporters/braintrust.test.ts7/7 tests green (5 existing + 2 new).
pnpm exec tsc -p tsconfig.json --noEmit — clean.

Linked issue

Refs #1405

…g() throws

A no-turn eval (e.g. `t.target.dispatchSchedule(...)` + `t.check` DB
assertions) legitimately produces `result.output === null` per the eval
API's own derivation. Braintrust's `validateAndSanitizeExperimentLogFullArgs`
rejects null/undefined output ("output must be specified"), and the throw
escaped `onEvalComplete`, killing the entire `eve eval` run — remaining
evals never executed and no artifacts were written, even though the
crashing eval itself passed all its gates.

Two fixes:

1. `output: result.result.output ?? ""` — coalesce null to an empty
   string so the SDK accepts it.
2. Wrap `experiment.log()` in try/catch — any reporter throw is logged
   to stderr but does not abort the run. The remaining evals still
   execute and artifacts are written.

Regression tests:
- `coalesces null output to empty string for no-turn evals` — drives an
  eval result with `output: null`, asserts `log` receives `output: ""`
- `survives a log() throw without aborting the run` — mocks `log` to
  throw, asserts no rethrow, then asserts a second call still reaches
  `log` (the next eval is not dropped)

Verified:
- `pnpm exec vitest run --config vitest.unit.config.ts src/evals/runner/reporters/braintrust.test.ts` — 7/7 tests green
- `pnpm exec tsc -p tsconfig.json --noEmit` — clean

Refs vercel#1405

Signed-off-by: Sarthak Singh <sarthak.singh@juspay.in>
@vercel

vercel Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

@iroiro147 is attempting to deploy a commit to the Vercel Team on Vercel.

A member of the Team first needs to authorize it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant