Skip to content

Default eval reporter drops suite hierarchy from describeEval output #40

@dcramer

Description

@dcramer

Summary

describeEval(...) is registering a real Vitest suite, but the default vitest-evals reporter output appears to flatten away the suite path and only prints the leaf test name during live runs.

Repro

import { describeEval } from "vitest-evals";

describeEval("new bottles", {
  data: async () => [{ name: "Jura 12-year-old Scotch Whisky", input: "..." }],
  task: async () => ({ result: "ok" }),
  scorers: [async () => ({ score: 1 })],
});

When listing tests with plain Vitest, the suite is present as expected. For example:

src/classifier.eval.test.ts > new bottles > Jura 12-year-old Scotch Whisky

But the live vitest-evals reporter output only shows the file and leaf test name, e.g.:

RUN  v4.1.4 /path/to/repo

✓ src/classifier.eval.test.ts > Jura 12-year-old Scotch Whisky [1.00]

The suite label (new bottles) is missing, which makes scenario-grouped eval runs much harder to read.

Likely cause

The reporter seems to format titles manually in a way that does not preserve the full suite path. In dist/reporter.js it logs:

  • test.module.task.name
  • this.getTestName(test.task, TEST_NAME_SEPARATOR)

That looks like it is not reconstructing the same full suite hierarchy that plain Vitest uses for vitest list.

Expected

Reporter output should include the same suite hierarchy users see from Vitest task listing, e.g.:

✓ src/classifier.eval.test.ts > new bottles > Jura 12-year-old Scotch Whisky [1.00]

Or at minimum preserve parent suite names for nested describe / describeEval blocks.

Environment

  • vitest-evals 0.8.0
  • vitest 4.1.4

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions