Skip to content

feat(eval): add Ragas-style context precision and recall metrics - #341

Open
Isha-Zaka wants to merge 1 commit into
codeforstartups:developmentfrom
Isha-Zaka:feat/issue-303-ragas-metrics
Open

Isha-Zaka wants to merge 1 commit into
codeforstartups:developmentfrom
Isha-Zaka:feat/issue-303-ragas-metrics

Conversation

@Isha-Zaka

Copy link
Copy Markdown
Collaborator

Description

Adds Ragas-style context precision and context recall metrics to the RAG evaluation framework, with model-layer judge support and batch evaluation for labeled examples.

Related issue

Fixes #303

Changes

  • Added Ragas-style context precision with rank-sensitive relevance scoring and context recall based on reference-answer claim coverage.
  • Added ChatModelJudge to support evaluation through Dynavec's existing ChatModel abstraction.
  • Extended evaluate_rag() and EvalRunner to support all four RAG evaluation metrics and batch evaluation with optional reference answers, with fake-judge test coverage.

Testing

Validation: 36 passed in tests/test_eval.py; full make run-ci passed; mypy passed on 81 source files; 1 snapshot passed; git diff --check passed.

  • Tests pass locally
  • Ruff checks pass
  • Documentation updated, if applicable

Checklist

  • My changes are focused and relevant to this pull request.
  • I have added or updated tests where appropriate.
  • I have reviewed my changes for unrelated modifications.
  • I have updated documentation where necessary.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Ragas-style metrics: faithfulness, answer relevance, context precision/recall

1 participant