Skip to content

MON-322 add reranking benchmark#574

Open
dommonkhouse wants to merge 5 commits into
zilliztech:mainfrom
dommonkhouse:dom/mon-322-memsearch-reranking-benchmark-implementation-plan
Open

MON-322 add reranking benchmark#574
dommonkhouse wants to merge 5 commits into
zilliztech:mainfrom
dommonkhouse:dom/mon-322-memsearch-reranking-benchmark-implementation-plan

Conversation

@dommonkhouse

Copy link
Copy Markdown

Summary

  • Add a reranking benchmark runner that compares plain hybrid MemSearch against optional ONNX reranking.
  • Add scored fixture coverage for hit@1, hit@3, hit@5, duplicate-source regressions, per-query diffs, warm-up metadata, and conservative recommendations.
  • Document benchmark-first rollout and rollback for reranker.model, including plain-mode isolation from global reranker config.

Test Plan

  • env -u MEMSEARCH_DIR uv run python -m pytest -q
  • uv run ruff check src tests scripts
  • uv run ruff format --check src tests scripts
  • uv run python scripts/benchmark_reranking.py --fixture-dir tests/fixtures/reranking --queries tests/fixtures/reranking/benchmark.json --collection ms_memsearch_ae2d4f9b --top-k 5 --reranker-model Alibaba-NLP/gte-reranker-modernbert-base --out /tmp/memsearch-reranking-fixture.json

Notes

  • No global MemSearch config was changed.
  • No deployment was performed.
  • MEMSEARCH_DIR is cleared for pytest because the local shell sets it to a live shared state directory; leaving it set causes unrelated maintenance tests to share state.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant