fix(evals): coalesce null output for Braintrust reporter and guard log() throws - #1530
Open
iroiro147 wants to merge 1 commit into
Open
fix(evals): coalesce null output for Braintrust reporter and guard log() throws#1530iroiro147 wants to merge 1 commit into
iroiro147 wants to merge 1 commit into
Conversation
…g() throws
A no-turn eval (e.g. `t.target.dispatchSchedule(...)` + `t.check` DB
assertions) legitimately produces `result.output === null` per the eval
API's own derivation. Braintrust's `validateAndSanitizeExperimentLogFullArgs`
rejects null/undefined output ("output must be specified"), and the throw
escaped `onEvalComplete`, killing the entire `eve eval` run — remaining
evals never executed and no artifacts were written, even though the
crashing eval itself passed all its gates.
Two fixes:
1. `output: result.result.output ?? ""` — coalesce null to an empty
string so the SDK accepts it.
2. Wrap `experiment.log()` in try/catch — any reporter throw is logged
to stderr but does not abort the run. The remaining evals still
execute and artifacts are written.
Regression tests:
- `coalesces null output to empty string for no-turn evals` — drives an
eval result with `output: null`, asserts `log` receives `output: ""`
- `survives a log() throw without aborting the run` — mocks `log` to
throw, asserts no rethrow, then asserts a second call still reaches
`log` (the next eval is not dropped)
Verified:
- `pnpm exec vitest run --config vitest.unit.config.ts src/evals/runner/reporters/braintrust.test.ts` — 7/7 tests green
- `pnpm exec tsc -p tsconfig.json --noEmit` — clean
Refs vercel#1405
Signed-off-by: Sarthak Singh <sarthak.singh@juspay.in>
Contributor
|
@iroiro147 is attempting to deploy a commit to the Vercel Team on Vercel. A member of the Team first needs to authorize it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A no-turn eval (e.g.
t.target.dispatchSchedule(...)+t.checkover DB state) legitimately producesresult.output === nullper the eval API's own derivation. Braintrust'svalidateAndSanitizeExperimentLogFullArgsrejects null/undefined output ("output must be specified"), and the throw escapedonEvalComplete, killing the entireeve evalrun — remaining evals never executed and no artifacts were written, even though the crashing eval itself passed all its gates.Two fixes in
BraintrustReporter.onEvalComplete:output: result.result.output ?? ""— coalesce null to an empty string so the Braintrust SDK accepts it.experiment.log()— any reporter throw is logged to stderr but does not abort the run. The remaining evals still execute and artifacts are written.Regression tests
Added to
braintrust.test.ts:coalesces null output to empty string for no-turn evals— drives an eval result withoutput: null, assertslogreceivesoutput: "".survives a log() throw without aborting the run— mockslogto throw"output must be specified", asserts no rethrow, then asserts a second call still reacheslog(proving the next eval is not dropped).pnpm exec vitest run --config vitest.unit.config.ts src/evals/runner/reporters/braintrust.test.ts— 7/7 tests green (5 existing + 2 new).pnpm exec tsc -p tsconfig.json --noEmit— clean.Linked issue
Refs #1405