Skip to content

bug in YAML in property testing_criteria of type Eval #466

@zijianhuang

Description

@zijianhuang

In

    Eval:
      type: object
      title: Eval
      description: |
        An Eval object with a data source config and testing criteria.
        An Eval represents a task to be done for your LLM integration.
        Like:
         - Improve the quality of my chatbot
         - See how well my chatbot handles customer support
         - Check if o3-mini is better at my usecase than gpt-4o
      properties:
        object:
          type: string
          enum:
            - eval
          default: eval
          description: The object type.
          x-stainless-const: true
        id:
          type: string
          description: Unique identifier for the evaluation.
        name:
          type: string
          description: The name of the evaluation.
          example: Chatbot effectiveness Evaluation
        data_source_config:
          type: object
          description: Configuration of data sources used in runs of the evaluation.
          oneOf:
            - $ref: "#/components/schemas/EvalCustomDataSourceConfig"
            - $ref: "#/components/schemas/EvalStoredCompletionsDataSourceConfig"
        testing_criteria:
          default: eval
          description: A list of testing criteria.
          type: array
          items:
            oneOf:
              - $ref: "#/components/schemas/EvalLabelModelGrader"
              - $ref: "#/components/schemas/EvalStringCheckGrader"
              - $ref: "#/components/schemas/EvalTextSimilarityGrader"
              - $ref: "#/components/schemas/EvalPythonGrader"
              - $ref: "#/components/schemas/EvalScoreModelGrader"

default: eval seems like a bug:

        testing_criteria:
          default: eval
          description: A list of testing criteria.
          type: array
          items:
            oneOf:
              - $ref: "#/components/schemas/EvalLabelModelGrader"
              - $ref: "#/components/schemas/EvalStringCheckGrader"
              - $ref: "#/components/schemas/EvalTextSimilarityGrader"
              - $ref: "#/components/schemas/EvalPythonGrader"
              - $ref: "#/components/schemas/EvalScoreModelGrader"

The property is of array type, and the array could contain one of the nominated types. So what is default value "eval"?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions