Braintrust reporter crashes the whole eval run when an eval produces no agent turn (output: null → "output must be specified")

[Jamie-Ellis]

Summary

An eval that never drives an agent turn — e.g. one that calls t.target.dispatchSchedule(...) and then asserts database state with t.check — completes with result.output === null (per the documented derivation: "the final turn's structured data when present, otherwise its last assistant message, or null when none was produced").

When the Braintrust reporter is active, BraintrustReporter.onEvalComplete forwards that value verbatim:

this.#n.log({ id: e.id, input: ..., output: e.result.output, ... })

braintrust's validateAndSanitizeExperimentLogFullArgs rejects null and undefined alike (isEmpty(event.output) → throw new Error("output must be specified")), and the throw escapes onEvalComplete, killing the entire eve eval run — remaining evals never execute and no artifacts are written, even though the crashing eval itself passed all its gates.

Repro

1. Any eval whose test performs no t.target message turn (schedule-dispatch assertions over DB state are the natural case).
2. Run eve eval <that-eval> with BRAINTRUST_API_KEY set.
3. The eval passes (✓ gates n/n), then the run exits 1 with output must be specified.

Observed on eve 0.27.12 and 0.27.13 (reporter dist byte-identical across the two); braintrust 3.9.0.

Expected

A no-turn eval is a legitimate shape (the eval API's own output doc anticipates null). The reporter should sanitize before logging — e.g. output: e.result.output ?? "" or omit/stub the field — and a reporter throw in onEvalComplete arguably shouldn't abort the run and drop the remaining evals + artifacts either.

Workaround we're using

Split invocation: keyed eve eval for turn-producing evals (posts the experiment), a second un-keyed run for the no-turn evals (console gates only). Retires when this is fixed.

[BlackishGreen33]

Hi, I'd like to work on a focused fix for this. I'll normalize null/undefined eval output at the Braintrust reporter boundary and add a regression test for evals that produce no agent turn. I'll keep the broader reporter error-handling policy out of scope.

[iroiro147]

Shipped a fix: #1530

Two changes in BraintrustReporter.onEvalComplete:

1. output: result.result.output ?? "" — coalesces null so the SDK accepts it
2. try/catch around experiment.log() — a reporter throw logs to stderr but doesn't abort the run

The workaround (split keyed/un-keyed eval runs) retires once this lands.