Summary
describeEval(...) is registering a real Vitest suite, but the default vitest-evals reporter output appears to flatten away the suite path and only prints the leaf test name during live runs.
Repro
import { describeEval } from "vitest-evals";
describeEval("new bottles", {
data: async () => [{ name: "Jura 12-year-old Scotch Whisky", input: "..." }],
task: async () => ({ result: "ok" }),
scorers: [async () => ({ score: 1 })],
});
When listing tests with plain Vitest, the suite is present as expected. For example:
src/classifier.eval.test.ts > new bottles > Jura 12-year-old Scotch Whisky
But the live vitest-evals reporter output only shows the file and leaf test name, e.g.:
RUN v4.1.4 /path/to/repo
✓ src/classifier.eval.test.ts > Jura 12-year-old Scotch Whisky [1.00]
The suite label (new bottles) is missing, which makes scenario-grouped eval runs much harder to read.
Likely cause
The reporter seems to format titles manually in a way that does not preserve the full suite path. In dist/reporter.js it logs:
test.module.task.name
this.getTestName(test.task, TEST_NAME_SEPARATOR)
That looks like it is not reconstructing the same full suite hierarchy that plain Vitest uses for vitest list.
Expected
Reporter output should include the same suite hierarchy users see from Vitest task listing, e.g.:
✓ src/classifier.eval.test.ts > new bottles > Jura 12-year-old Scotch Whisky [1.00]
Or at minimum preserve parent suite names for nested describe / describeEval blocks.
Environment
vitest-evals 0.8.0
vitest 4.1.4
Summary
describeEval(...)is registering a real Vitest suite, but the defaultvitest-evalsreporter output appears to flatten away the suite path and only prints the leaf test name during live runs.Repro
When listing tests with plain Vitest, the suite is present as expected. For example:
But the live
vitest-evalsreporter output only shows the file and leaf test name, e.g.:The suite label (
new bottles) is missing, which makes scenario-grouped eval runs much harder to read.Likely cause
The reporter seems to format titles manually in a way that does not preserve the full suite path. In
dist/reporter.jsit logs:test.module.task.namethis.getTestName(test.task, TEST_NAME_SEPARATOR)That looks like it is not reconstructing the same full suite hierarchy that plain Vitest uses for
vitest list.Expected
Reporter output should include the same suite hierarchy users see from Vitest task listing, e.g.:
Or at minimum preserve parent suite names for nested
describe/describeEvalblocks.Environment
vitest-evals0.8.0vitest4.1.4