-
Notifications
You must be signed in to change notification settings - Fork 465
Open
Description
In
Eval:
type: object
title: Eval
description: |
An Eval object with a data source config and testing criteria.
An Eval represents a task to be done for your LLM integration.
Like:
- Improve the quality of my chatbot
- See how well my chatbot handles customer support
- Check if o3-mini is better at my usecase than gpt-4o
properties:
object:
type: string
enum:
- eval
default: eval
description: The object type.
x-stainless-const: true
id:
type: string
description: Unique identifier for the evaluation.
name:
type: string
description: The name of the evaluation.
example: Chatbot effectiveness Evaluation
data_source_config:
type: object
description: Configuration of data sources used in runs of the evaluation.
oneOf:
- $ref: "#/components/schemas/EvalCustomDataSourceConfig"
- $ref: "#/components/schemas/EvalStoredCompletionsDataSourceConfig"
testing_criteria:
default: eval
description: A list of testing criteria.
type: array
items:
oneOf:
- $ref: "#/components/schemas/EvalLabelModelGrader"
- $ref: "#/components/schemas/EvalStringCheckGrader"
- $ref: "#/components/schemas/EvalTextSimilarityGrader"
- $ref: "#/components/schemas/EvalPythonGrader"
- $ref: "#/components/schemas/EvalScoreModelGrader"
default: eval seems like a bug:
testing_criteria:
default: eval
description: A list of testing criteria.
type: array
items:
oneOf:
- $ref: "#/components/schemas/EvalLabelModelGrader"
- $ref: "#/components/schemas/EvalStringCheckGrader"
- $ref: "#/components/schemas/EvalTextSimilarityGrader"
- $ref: "#/components/schemas/EvalPythonGrader"
- $ref: "#/components/schemas/EvalScoreModelGrader"
The property is of array type, and the array could contain one of the nominated types. So what is default value "eval"?
Metadata
Metadata
Assignees
Labels
No labels