📊 ragbench

ragbench is a lightweight RAG benchmark and RAG evaluation framework.

Benchmark any retrieval-augmented generation (RAG) pipeline with synthetic datasets and deterministic metrics — no LLM API required for the core. Plug in your retriever and generator (LangChain, LlamaIndex, Haystack, or hand-rolled), run over a built-in dataset, and get a reproducible scorecard of retrieval and generation quality.

Built incrementally as a medium-complexity project. Retrieval + generation metrics and synthetic datasets ship first; an optional LLM-as-judge backend is planned.

Why ragbench?

• No API costs for core evals — deterministic metrics (Precision@K, Recall@K, MRR, NDCG, AnswerOverlap) run entirely locally. Only pay for LLM calls when you opt into the judge backend.
• Framework-agnostic — your pipeline is just (retriever, generator) callables. Works with LangChain, LlamaIndex, Haystack, or hand-rolled code.
• Reproducible by design — versioned synthetic datasets, no network calls in the core path. Same dataset, same scorecard, every time.
• Run-diffing built in — A/B compare two pipelines or configs head-to-head with diff_runs.

Features

• 🧪 Synthetic datasets — a 10-query factual set + multi-hop set, with ground-truth relevance, so you can benchmark without a private corpus
• 📏 Deterministic metrics — Precision@K, Recall@K, MRR, NDCG@K (retrieval) + AnswerOverlap, Faithfulness-lite, CitationCoverage (generation)
• 🔌 Framework-agnostic — your pipeline is just (retriever, generator) callables
• 📋 Scorecards — per-metric means ± stdev, overall weighted score, pass-rate
• 🔁 Reproducible — versioned datasets, no network calls in the core
• 🚫 Zero hard dependencies — pure stdlib; optional openai for the planned judge backend

Quick Start

pip install ragbench

Usage

Benchmark a pipeline

from ragbench import run_benchmark, scorecard
from ragbench.datasets import factual
from ragbench.report import render_scorecard
from ragbench.types import RetrievedDoc, Answer

syn = factual()                       # dataset + companion corpus
docs = syn.as_retrieved_docs()

def retriever(query: str) -> list[RetrievedDoc]:
    ...  # your retrieval logic, returns RetrievedDoc lists

def generator(query: str, docs: list[RetrievedDoc]) -> Answer:
    ...  # your LLM call, returns an Answer

run = run_benchmark(syn.dataset, retriever, generator)
print(render_scorecard(scorecard(run)))

Example output:

ragbench — factual (10 queries, 0.01s)
  overall: 0.940   pass-rate: 100.0%
  metrics:
    answer_overlap        1.000 ±0.000  ████████████████████
    citation_coverage     1.000 ±0.000  ████████████████████
    faithfulness_lite     0.980 ±0.040  ███████████████████
    mrr                   1.000 ±0.000  ████████████████████
    ndcg@k                1.000 ±0.000  ████████████████████
    precision@k           0.200 ±0.000  ████
    recall@k              1.000 ±0.000  ████████████████████

Datasets

Dataset	Queries	Notes
factual()	10	single-doc factual Q&A across 5 categories
multi_hop(n)	n	two-doc evidence, tests full-evidence recall

Metrics

Retrieval (need relevant_doc_ids ground truth): Precision@K, Recall@K, MRR, NDCG@K.

Generation (no LLM judge): AnswerOverlap (token Jaccard vs expected), Faithfulness-lite (fraction of answer tokens supported by context — a hallucination proxy), CitationCoverage (cited docs that were actually retrieved).

LLM-as-judge (optional): FaithfulnessJudge + AnswerRelevanceJudge — a pluggable JudgeBackend scores answers. Ships with a deterministic MockJudge (no network, real heuristics) and an LLMJudge (optional openai dep). Swap the backend, keep the metric.

CLI

ragbench datasets                       # list datasets
ragbench metrics --judge mock           # list the metric suite
ragbench run --dataset factual          # run reference pipeline, print scorecard
ragbench run --dataset adversarial      # hard-negative distractors (precision stress)
ragbench run --dataset long-tail        # head + tail queries (recall on rare queries)
ragbench run --pipeline mypipe.py       # benchmark YOUR retriever+generator
ragbench run --judge mock --json        # add judge metrics, JSON output
ragbench run --judge llm --model gpt-4o-mini   # real LLM judge (needs OPENAI_API_KEY)
ragbench compare --dataset factual      # A/B diff: good vs weak retriever

Benchmark your own pipeline

Write a mypipe.py exposing retriever and generator, then:

ragbench run --pipeline mypipe.py --dataset factual --json

# mypipe.py
from ragbench.types import RetrievedDoc, Answer

def retriever(query: str) -> list[RetrievedDoc]:
    ...  # your retrieval logic

def generator(query: str, docs: list[RetrievedDoc]) -> Answer:
    ...  # your LLM call

Run-diffing (A/B)

Compare two pipelines or configs head-to-head:

from ragbench import run_benchmark, diff_runs, render_diff

run_a = run_benchmark(dataset, retriever_a, generator)
run_b = run_benchmark(dataset, retriever_b, generator)
print(render_diff(diff_runs(run_a, run_b, "pipe-a", "pipe-b")))

Roadmap

• LLM-as-judge backend (faithfulness, answer-relevance) — optional openai dep
• CLI: ragbench run --dataset factual
• Run diffing: compare two pipelines head-to-head
• More synthetic datasets (adversarial, long-tail)
• Pluggable user pipelines (--pipeline mypipe.py)

Development

git clone https://github.com/alvabillwu/ragbench.git
cd ragbench
pip install -e ".[dev]"
pytest -v
python examples/benchmark.py
ragbench run --judge mock

🔄 Alternatives

An honest comparison with related RAG evaluation frameworks:

Tool	Core metrics need an LLM?	Dependencies	Best for
ragbench	No — deterministic core (optional LLM judge)	Zero hard deps (stdlib)	Fast, reproducible, offline retrieval + generation scoring in CI
RAGAS	Yes — most metrics use an LLM + embeddings	Heavier (LLM/embedding providers)	Rich, research-grade RAG metrics when calling a model is fine
TruLens	Yes — feedback functions call models	Heavier	Instrumented eval + observability of live RAG apps
DeepEval	Often — LLM-based metrics	Moderate	Pytest-style LLM/RAG test assertions

When to use ragbench: you want deterministic, no-network retrieval and generation metrics (Precision@K, Recall@K, MRR, NDCG, AnswerOverlap) that produce the same scorecard every run — ideal for CI and A/B pipeline diffs. Reach for RAGAS, TruLens, or DeepEval when you specifically want LLM-graded metrics and don't mind the API calls; RAGAS in particular supports custom/local models if you'd rather not hit a hosted API. ragbench also ships an optional LLM judge if you want both worlds.

🔗 Ecosystem

Part of the alvabillwu AI/LLM DevTools suite:

• synthqa — Synthetic QA evaluation dataset generator
• embeddings-bench — Embedding model benchmarking toolkit
• judgekit — Reusable LLM-as-judge harness for evaluation
• eval-suite — Unified evaluation suite for LLM outputs
• tokencost — LLM token cost calculator and visualizer

License

MIT © alvabillwu

---

⭐ Like this? Star the repo to support open-source RAG evaluation tooling!