RFC: chunk_rag as method-baseline eval vs graph_rag

[Frankie-Xu]

RFC (waiting on maintainer)

v0.9.0 release notes (docs/en/news.md) introduce chunk_rag as the zero-extraction chunk-retrieval baseline for corpus Q&A and method benchmarking. There is no integration eval that compares it to graph_rag.

Question

Would you accept tests under tests/integration that:

1. mock embeddings / LLM where possible
2. optionally run live recall@k
3. compare chunk_rag vs graph_rag on examples/en/tesla.md

Out of scope until this RFC is accepted

• No eval harness in this round

[Frankie-Xu]

Follow-up: still waiting on a maintainer call on this RFC.

chunk_rag is documented as the zero-extraction baseline for method benchmarking, but there is still no integration eval against graph_rag (mocked embeddings/LLM and/or live recall@k on examples/en/tesla.md).

No patch attached. Will not add an eval harness until you confirm the scope.

[yifanfeng97]

Agreed with the framing — chunk_rag exists precisely to be the zero-extraction baseline that graph methods should be measured against (and it shares the full provenance machinery, so comparisons are apples-to-apples on the same corpus).

What would make this concrete, roughly in increasing scope:

1. Docs page — "Choosing a method: baseline vs structured", showing when the extra cost of graph extraction pays off (multi-hop questions, relationship-heavy queries) vs when chunks are enough (lookup-style questions).
2. Example script — same corpus fed to chunk_rag and graph_rag, identical questions, side-by-side answers + token cost + latency. No framework needed.
3. Eval harness — a small benchmark set (QA pairs + expected source spans) with retrieval hit-rate metrics. This one needs care: retrieval metrics on hash-like embedding models in CI would be meaningless, so it would have to be an opt-in script against a real embedder, not a unit test.

Happy to review a proposal or PR for (1)/(2); (3) probably deserves its own RFC with the metric definitions spelled out.

[Frankie-Xu]

Thanks for the scoped reply. Received — this round will not add the method-choice docs page, comparison script, or eval harness. Happy to pick up (1)/(2) in a later pass if you still want them.

[Frankie-Xu]

Picking up (1) and (2) in this pass: a "Choosing a method: baseline vs structured" docs page, plus an example script on the Tesla corpus. Still not adding the eval harness — that stays a separate RFC as you asked.