Expand "input" type definition to support more than strings

[davidturissini]

We are building an image generation pipeline and we want to score images outputted by LLMs. Our eval inputs are not strings, they are product/sku ids that are numeric. Currently, we have to cast our sku ids to strings. This isn't the end of the world, but I believe that more expressive types here would allow for more complicated eval scenarios beyond just text in and out.

This is a type error today:

describeEval('my-eval', {
        data: async () => {
          return [
            {
              input: 123 <-- number
            },

[dcramer]

Somehow I never see notifications on here so sorry for the late response!

I think in general the approach to inputs in the library needs improved. I did a bit of this when adding the tool call scoring but I’m also not totally sure we’ll see annoying issues here making input type more flexible (eg “any”). Might just be the responsibility of the scorers to deal with it, but I’m not totally sure we can easily retain type safety?

[davidturissini]

This does introduce breaking changes, but this changeset does retain types all the way through task and scorer functions while also requiring that the user supply a string "input".

https://github.com/getsentry/vitest-evals/compare/main...davidturissini:dt/describe-eval-type-updates?expand=1

I'm not sure requiring an "input" makes as much sense with this change because the user would have the ability to call their input whatever.

How are you thinking about inputs wrt this library?

[zirkelc]

I agree that having structured input rather than a string would be useful.

Here's a quick hack to get it working with an object by overriding the toString function:

const createInput = (name: string, data: InputData) =>
  Object.assign(data, { toString: () => name });

describeEval(`Identify Language`, {
  data: async () => [
    {
      input: createInput(
        'should respond in English', 
        { messages: [`Hi`], language: ``, country: `` },
      ),
      expected: `en`,
    },
  ],
  task: async (input) => {
    const { messages, language, country } = input as unknown as InputData;

    // ...
  },
  scorers: [  ],
  threshold: 1.0,
});