Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: Add link to RAGAS metrics page #2019

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

docs: Add link to RAGAS metrics page #2019

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b4896bd
docs: Add link to RAGAS metrics page
ssbushi Feb 18, 2025
a82c0de
Update evaluation.md
ssbushi Feb 18, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions docs/evaluation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -252,6 +252,9 @@ given prompt
* Maliciousness -- Measures whether the generated output intends to deceive,
harm, or exploit

For more details on how each metric is computed, please refer to the
[RAGAS metrics documentation](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/){: .external}.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looking at the page, there are many more metrics listed there... do we want to link to specific metrics instead?
Ex: https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/faithfulness/

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought of that, but RAGAS change their metrics frequently, I was afraid we might have to keep up with them so I used the parent page instead.

I can switch to using specific metrics if you feel that's preferred.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if they change the metrics frequently then it means that whatever we have may not be up-to-date, right? what's the point of linking to their docs if it may or may not be accurate?

Copy link
Contributor Author

@ssbushi ssbushi Feb 19, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason we should link is because they have detailed explanation on how a metric is computed. Some of the feedback we received for evals is a scenario like: 'user expects relevancy to be 1.0 but it is only 0.5, why is that?'

The reason I am hesitant to link to individual docs for each metric is because RAGAS seem to change their metrics collection (either removing them or renaming them). E.g.: ANSWER_RELEVANCY in genkitEvals is now called "Response Relevancy" in RAGAS.

Catching up to RAGAS is a different problem altogether.

You are right though... taking them to a top level directory does not guarantee that the metric details would be available. The way I see it, both approaches have their cons.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alternatively, should we just bring the explanations of the metrics into our docs so we don't need to link our to RAGAS docs?

I believe our metrics are already just "RAGAS-inspired", so it's okay if we deviate as long as we explain what they mean well.


### Evaluator plugins

Genkit supports additional evaluators through plugins, like the Vertex Rapid
Expand Down
Loading