Hybrid Search Engine

A modular hybrid retrieval system for enterprise-scale document search, combining dense semantic retrieval and sparse keyword retrieval with multi-stage reranking pipelines.

Overview

Hybrid Search Engine is a retrieval-focused search infrastructure project designed to efficiently search large document collections using modern information retrieval techniques. The system combines semantic vector-based retrieval with traditional sparse keyword retrieval to improve both retrieval robustness and ranking quality.

The architecture implements:

• Dense semantic retrieval using SentenceTransformer embeddings and FAISS vector indexing
• Sparse keyword retrieval using BM25
• Hybrid retrieval using Reciprocal Rank Fusion (RRF)
• Multi-stage retrieval pipelines with CrossEncoder reranking
• Metadata-aware filtering for enterprise-style document retrieval
• Modular ingestion and retrieval orchestration

The project focuses on retrieval systems engineering and ranking pipelines rather than frontend application development.

---

System Architecture

Retrieval Pipeline

Documents
   ↓
Chunking Pipeline
   ↓
SentenceTransformer Bi-Encoder Embeddings
   ↓
FAISS Dense Vector Index

Documents
   ↓
BM25 Sparse Index

User Query
   ↓
Dense Retrieval + Sparse Retrieval
   ↓
Reciprocal Rank Fusion (RRF)
   ↓
Candidate Generation
   ↓
CrossEncoder Reranking
   ↓
Final Ranked Results

---

Features

Dense Semantic Retrieval

• SentenceTransformer-based semantic embeddings
• FAISS vector indexing using IndexFlatIP
• Cosine similarity-based nearest neighbor retrieval
• Embedding normalization for semantic similarity search

Sparse Keyword Retrieval

• BM25 ranking for exact keyword-based retrieval
• Frequency-aware term scoring
• Improved retrieval robustness for keyword-heavy queries

Hybrid Retrieval

• Combines dense and sparse retrieval pipelines
• Reciprocal Rank Fusion (RRF) for hybrid ranking
• Improved recall and ranking stability

CrossEncoder Reranking

• Multi-stage retrieval architecture
• Transformer-based reranking for improved precision
• Joint query-document relevance modeling
• Candidate reranking after hybrid retrieval

Metadata-Aware Filtering

Supports filtering by:

• Category
• Source
• Document metadata

Modular Architecture

• Modular ingestion pipeline
• Extensible retrieval orchestration
• Separated indexing and retrieval components
• Easy integration of additional retrieval strategies

---

Project Structure

Hybrid-Search-Engine/
│
├── data/
│   └── raw_docs/
│
├── ingest/
│   ├── chunker.py
│   ├── embedder.py
│   └── loader.py
│
├── search/
│   ├── bm25_index.py
│   ├── reranker.py
│   ├── retriever.py
│   └── vector_index.py
│
├── test_retrieval.py
├── requirements.txt
└── README.md

---

Core Components

chunker.py

Splits large documents into overlapping chunks to improve retrieval granularity and semantic coverage.

embedder.py

Generates semantic embeddings using SentenceTransformer bi-encoder models.

loader.py

Loads documents, applies chunking, generates embeddings, and constructs retrieval records.

vector_index.py

Implements dense vector indexing and nearest neighbor search using FAISS.

bm25_index.py

Implements sparse retrieval using BM25 ranking.

retriever.py

Implements:

• Dense retrieval
• Sparse retrieval
• Hybrid retrieval
• Reciprocal Rank Fusion
• Metadata filtering
• CrossEncoder reranking pipelines

reranker.py

Implements transformer-based CrossEncoder reranking for final ranking refinement.

---

Technologies Used

• Python
• SentenceTransformers
• FAISS
• BM25
• NumPy
• CrossEncoder models

---

Installation

Clone Repository

git clone <repository-url>
cd Hybrid-Search-Engine

Create Virtual Environment

python -m venv venv

Activate Environment

Windows

venv\Scripts\activate

Linux / macOS

source venv/bin/activate

Install Dependencies

pip install -r requirements.txt

---

Running the Project

Place .txt documents inside:

data/raw_docs/

Run:

python test_retrieval.py

The system will:

1. Load and preprocess documents
2. Chunk documents
3. Generate semantic embeddings
4. Build FAISS vector index
5. Build BM25 sparse index
6. Perform hybrid retrieval
7. Apply CrossEncoder reranking
8. Return final ranked search results

---

Example Retrieval Workflow

Query

semantic search systems

Pipeline

1. Query embedding generation
2. Dense semantic retrieval using FAISS
3. Sparse keyword retrieval using BM25
4. Hybrid fusion using RRF
5. Candidate reranking using CrossEncoder
6. Final ranked output generation

---

Retrieval Design Decisions

Why Hybrid Retrieval?

Dense semantic retrieval improves semantic understanding, while sparse retrieval improves exact keyword matching. Combining both improves retrieval robustness and recall.

Why Reciprocal Rank Fusion?

RRF provides a lightweight and effective way to combine rankings from heterogeneous retrieval systems without requiring score normalization.

Why CrossEncoder Reranking?

Bi-encoder retrieval is scalable but less precise because queries and documents are encoded independently. CrossEncoders jointly model query-document interactions, significantly improving ranking precision.

Why Multi-Stage Retrieval?

CrossEncoder inference is computationally expensive. Therefore, the system first retrieves a smaller high-recall candidate set using efficient retrieval methods before applying expensive reranking.

---

Future Improvements

Potential future extensions include:

• Persistent vector indexes
• PDF and DOCX ingestion
• ANN indexing strategies (IVF, HNSW)
• Retrieval evaluation metrics
• REST API integration
• Distributed vector storage

---

Learning Outcomes

This project explores core concepts in:

• Information Retrieval (IR)
• Semantic Search
• Dense and Sparse Retrieval
• Vector Databases
• Ranking Systems
• Retrieval Pip