Open
Conversation
Centralises the resolve → branch → construct pattern for local HF embedding models (VL and non-VL) that was duplicated across batch, inprocess, fused, gpu_pool, recall, retriever, and text_embed code paths into a single `create_local_embedder` factory function. Made-with: Cursor
Extracts duplicated LanceDB row-building, schema definition, and table-creation logic from batch.py and inprocess.py into a shared ingest_modes/lancedb_utils.py module. Made-with: Cursor
- Remove unused Path import and unused _extract_* aliases from inprocess.py - Remove unused pytest import from test_lancedb_utils.py - Apply black formatting to set literal and DataFrame constructor Made-with: Cursor
…import The ingest_modes __init__.py eagerly imports batch/fused/inprocess/online which pull in ray, torch, etc. Pre-populate sys.modules with MagicMock stubs so lancedb_utils tests can run in lightweight CI without those deps. Made-with: Cursor
Centralises gold_to_doc_page, hit_key_and_distance, estimate_processed_pages, and print_pages_per_second that were duplicated across batch, inprocess, online, and fused pipeline examples. Fixes broken imports in fused_pipeline.py that referenced non-existent functions in batch_pipeline.py. Made-with: Cursor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Consolidates duplicated recall-evaluation helpers and pipeline utility functions that were copy-pasted across
batch_pipeline.py,inprocess_pipeline.py,online_pipeline.py, andfused_pipeline.pyinto two shared modules.recall/core.py— Addsgold_to_doc_page()andhit_key_and_distance()alongside the existing recall functions. These were independently defined (with minor variations) in three pipeline scripts.examples/common.py(new) — Addsestimate_processed_pages()andprint_pages_per_second(), previously duplicated inbatch_pipeline.pyandinprocess_pipeline.py.fused_pipeline.pynow importsgold_to_doc_page,hit_key_and_distance, andis_hit_at_kfromrecall/core.pyandestimate_processed_pages/print_pages_per_secondfromexamples/common.py, instead of reaching intobatch_pipeline's private functions.batch_pipeline._ensure_lancedb_tablenow useslancedb_schema()fromlancedb_utils(PR2) instead of an inline 10-field PyArrow schema definition.hf_model_registry.py.Net diff: +134 / −227 lines (~93 lines removed)
Stacked on
consolidate/pr2-lancedb-utilsFiles changed
nemo_retriever/src/nemo_retriever/recall/core.pygold_to_doc_page()andhit_key_and_distance()nemo_retriever/src/nemo_retriever/examples/common.pyestimate_processed_pages(),print_pages_per_second()nemo_retriever/src/nemo_retriever/examples/batch_pipeline.pylancedb_schema()in_ensure_lancedb_tablenemo_retriever/src/nemo_retriever/examples/inprocess_pipeline.pyrecall/coreandexamples/commonnemo_retriever/src/nemo_retriever/examples/online_pipeline.pyrecall/corenemo_retriever/src/nemo_retriever/examples/fused_pipeline.pybatch_pipeline._*private imports to shared public APIsnemo_retriever/src/nemo_retriever/utils/hf_model_registry.pynvidia/llama-nemotron-embed-1b-v2entrynemo_retriever/tests/test_create_local_embedder.pynemo_retriever/tests/test_lancedb_utils.pyTest plan
test_create_local_embedder,test_lancedb_utils,test_multimodal_embed)inprocess_pipeline,batch_pipeline,online_pipeline, andfused_pipelineexecute end-to-end with recall evaluation enabledgold_to_doc_page/hit_key_and_distanceproduce identical results to the removed per-pipeline versionsestimate_processed_pages/print_pages_per_secondoutput matches previous behaviorChecklist