fix(js/vertexai): dedupe, improve types, fix rouge #3592
+72
−56
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
10kc-emilyk
commented
Sep 17, 2025
Comment on lines
-20
to
-50
|
|
||
| /** | ||
| * Vertex AI Evaluation metrics. See API documentation for more information. | ||
| * https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/evaluation#parameter-list | ||
| */ | ||
| export enum VertexAIEvaluationMetricType { | ||
| // Update genkit/docs/plugins/vertex-ai.md when modifying the list of enums | ||
| BLEU = 'BLEU', | ||
| ROUGE = 'ROUGE', | ||
| FLUENCY = 'FLEUNCY', | ||
| SAFETY = 'SAFETY', | ||
| GROUNDEDNESS = 'GROUNDEDNESS', | ||
| SUMMARIZATION_QUALITY = 'SUMMARIZATION_QUALITY', | ||
| SUMMARIZATION_HELPFULNESS = 'SUMMARIZATION_HELPFULNESS', | ||
| SUMMARIZATION_VERBOSITY = 'SUMMARIZATION_VERBOSITY', | ||
| } | ||
|
|
||
| /** | ||
| * Evaluation metric config. Use `metricSpec` to define the behavior of the metric. | ||
| * The value of `metricSpec` will be included in the request to the API. See the API documentation | ||
| * for details on the possible values of `metricSpec` for each metric. | ||
| * https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/evaluation#parameter-list | ||
| */ | ||
| export type VertexAIEvaluationMetricConfig = { | ||
| type: VertexAIEvaluationMetricType; | ||
| metricSpec: any; | ||
| }; | ||
|
|
||
| export type VertexAIEvaluationMetric = | ||
| | VertexAIEvaluationMetricType | ||
| | VertexAIEvaluationMetricConfig; |
There was a problem hiding this comment.
this was duplicated from types.ts
10kc-emilyk
commented
Sep 24, 2025
| // Update genkit/docs/plugins/vertex-ai.md when modifying the list of enums | ||
| BLEU = 'BLEU', | ||
| ROUGE = 'ROUGE', | ||
| FLUENCY = 'FLEUNCY', |
There was a problem hiding this comment.
Also, note that this currently has a typo in the value:
FLEUNCY should be
FLUENCY
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
While investigating the appropriate types for vertex evaluators, I'm adding them here too, and discovered an incorrect type used for the ROUGE score evaluator. Also there was some code duplication with the types.
Checklist (if applicable):
Docs updated (updated docs or a docs bug required)