Skip to content

Commit d5dcd9c

Browse files
xav-dbxav-db
authored
Merge pull request #34 from HelixDB/rerank
adding rerank
2 parents 36c6526 + 686d7ab commit d5dcd9c

File tree

5 files changed

+1865
-0
lines changed

5 files changed

+1865
-0
lines changed

‎docs.json‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -104,6 +104,14 @@
104104
"icon": "share-nodes"
105105
},
106106
"documentation/hql/keyword_search",
107+
{
108+
"group": "Rerankers",
109+
"expanded": false,
110+
"pages": [
111+
"documentation/hql/rerankers/overview"
112+
],
113+
"icon": "arrows-up-down"
114+
},
107115
{
108116
"group": "Traversals",
109117
"expanded": false,

‎documentation/hql/rerankers/overview copy.mdx‎

Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
---
2+
title: "Rerankers Overview"
3+
description: "Improve search result quality by reordering results after initial retrieval."
4+
icon: "arrows-up-down"
5+
---
6+
7+
## What are Rerankers?
8+
9+
Rerankers are powerful post-processing operations that improve the quality and diversity of search results by reordering them after the initial retrieval phase. They enable you to:
10+
11+
- Combine results from multiple search strategies (hybrid search)
12+
- Reduce redundancy by diversifying results
13+
- Optimize the relevance-diversity trade-off
14+
- Improve the overall user experience of your search application
15+
16+
## When to Use Rerankers
17+
18+
Apply rerankers in your query pipeline when you need to:
19+
20+
- **Merge multiple search methods**: Combine vector search with BM25 keyword search, or merge results from multiple vector searches
21+
- **Diversify results**: Eliminate near-duplicate content and show varied perspectives
22+
- **Optimize ranking**: Fine-tune the balance between relevance and variety based on your use case
23+
- **Improve search quality**: Leverage sophisticated ranking algorithms without changing your underlying search infrastructure
24+
25+
## Available Rerankers
26+
27+
HelixQL provides two powerful reranking strategies:
28+
29+
### RerankRRF (Reciprocal Rank Fusion)
30+
31+
A technique for combining multiple ranked lists without requiring score calibration. Perfect for hybrid search scenarios where you want to merge results from different search methods.
32+
33+
```rust
34+
::RerankRRF() // Uses default k=60
35+
::RerankRRF(k: 30.0) // Custom k parameter
36+
```
37+
38+
[Learn more about RerankRRF →](/documentation/hql/rerankers/rerank-rrf)
39+
40+
### RerankMMR (Maximal Marginal Relevance)
41+
42+
A diversification technique that balances relevance with diversity to reduce redundancy. Ideal when you want to show varied results instead of similar or duplicate content.
43+
44+
```rust
45+
::RerankMMR(lambda: 0.7) // Default cosine distance
46+
::RerankMMR(lambda: 0.5, distance: "euclidean") // Custom distance metric
47+
```
48+
49+
[Learn more about RerankMMR →](/documentation/hql/rerankers/rerank-mmr)
50+
51+
## Basic Usage Pattern
52+
53+
RRF Usage:
54+
```rust focus=3
55+
QUERY SearchDocuments(query_vec: [F64]) =>
56+
results <- SearchV<Document>(query_vec, 100)
57+
::RerankRRF() // Apply reranking
58+
::RANGE(0, 10) // Get top 10 results
59+
RETURN results
60+
```
61+
62+
MMR Usage:
63+
64+
```rust focus=3
65+
QUERY SearchDocuments(query_vec: [F64]) =>
66+
results <- SearchV<Document>(query_vec, 100)
67+
::RerankMMR(lambda: 0.7) // Apply reranking
68+
::RANGE(0, 10) // Get top 10 results
69+
RETURN results
70+
```
71+
72+
## Chaining Rerankers
73+
74+
You can chain multiple rerankers together for complex result optimization:
75+
76+
```rust focus={3-4}
77+
QUERY AdvancedSearch(query_vec: [F64]) =>
78+
results <- SearchV<Document>(query_vec, 150)
79+
::RerankRRF(k: 60) // First: combine multiple rankings
80+
::RerankMMR(lambda: 0.6) // Then: diversify results
81+
::RANGE(0, 10)
82+
RETURN results
83+
```
84+
85+
## Best Practices
86+
87+
1. **Retrieve more results initially**: Fetch 100-200 candidates to give rerankers sufficient options to work with
88+
2. **Apply rerankers before RANGE**: Rerank first, then limit the number of results returned
89+
3. **Choose the right reranker**: Use RRF for combining searches, MMR for diversification
90+
4. **Test with your data**: Experiment with different parameters to find what works best for your use case
91+
92+
## Common Patterns
93+
94+
```rust
95+
// Pattern 1: Simple diversification
96+
SearchV<Document>(vec, 100)::RerankMMR(lambda: 0.7)::RANGE(0, 10)
97+
98+
// Pattern 2: Hybrid search fusion
99+
SearchV<Document>(vec, 100)::RerankRRF()::RANGE(0, 10)
100+
101+
// Pattern 3: Fusion + diversification
102+
SearchV<Document>(vec, 150)::RerankRRF()::RerankMMR(lambda: 0.6)::RANGE(0, 10)
103+
```
104+
105+
## Next Steps
106+
107+
Explore the detailed documentation for each reranker:
108+
109+
- [RerankRRF (Reciprocal Rank Fusion)](/documentation/hql/rerankers/rerank-rrf)
110+
- [RerankMMR (Maximal Marginal Relevance)](/documentation/hql/rerankers/rerank-mmr)

‎documentation/hql/rerankers/overview.mdx‎

Lines changed: 99 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,99 @@
1+
---
2+
title: "Rerankers Overview"
3+
description: "Improve search result quality by reordering results after initial retrieval."
4+
icon: "arrows-up-down"
5+
---
6+
7+
## What are Rerankers?
8+
9+
Rerankers are powerful post-processing operations that improve the quality and diversity of search results by reordering them after the initial retrieval phase. They enable you to:
10+
11+
- Combine results from multiple search strategies (hybrid search)
12+
- Reduce redundancy by diversifying results
13+
- Optimize the relevance-diversity trade-off
14+
- Improve the overall user experience of your search application
15+
16+
## When to Use Rerankers
17+
18+
Apply rerankers in your query pipeline when you need to:
19+
20+
- **Merge multiple search methods**: Combine vector search with BM25 keyword search, or merge results from multiple vector searches
21+
- **Diversify results**: Eliminate near-duplicate content and show varied perspectives
22+
- **Optimize ranking**: Fine-tune the balance between relevance and variety based on your use case
23+
- **Improve search quality**: Leverage sophisticated ranking algorithms without changing your underlying search infrastructure
24+
25+
## Available Rerankers
26+
27+
HelixQL provides two powerful reranking strategies:
28+
29+
### RerankRRF (Reciprocal Rank Fusion)
30+
31+
A technique for combining multiple ranked lists without requiring score calibration. Perfect for hybrid search scenarios where you want to merge results from different search methods.
32+
33+
```rust
34+
::RerankRRF // Uses default k=60
35+
::RerankRRF(k: 30.0) // Custom k parameter
36+
```
37+
38+
### RerankMMR (Maximal Marginal Relevance)
39+
40+
A diversification technique that balances relevance with diversity to reduce redundancy. Ideal when you want to show varied results instead of similar or duplicate content.
41+
42+
```rust
43+
::RerankMMR(lambda: 0.7)
44+
```
45+
46+
47+
## Basic Usage Pattern
48+
49+
RRF Usage:
50+
```rust focus=3
51+
QUERY SearchDocuments(query_vec: [F64]) =>
52+
results <- SearchV<Document>(query_vec, 100)
53+
::RerankRRF // Apply reranking
54+
::RANGE(0, 10) // Get top 10 results
55+
RETURN results
56+
```
57+
58+
MMR Usage:
59+
60+
```rust focus=3
61+
QUERY SearchDocuments(query_vec: [F64]) =>
62+
results <- SearchV<Document>(query_vec, 100)
63+
::RerankMMR(lambda: 0.7) // Apply reranking
64+
::RANGE(0, 10) // Get top 10 results
65+
RETURN results
66+
```
67+
68+
## Chaining Rerankers
69+
70+
You can chain multiple rerankers together for complex result optimization:
71+
72+
```rust focus={3-4}
73+
QUERY AdvancedSearch(query_vec: [F64]) =>
74+
results <- SearchV<Document>(query_vec, 150)
75+
::RerankRRF(k: 60) // First: combine multiple rankings
76+
::RerankMMR(lambda: 0.6) // Then: diversify results
77+
::RANGE(0, 10)
78+
RETURN results
79+
```
80+
81+
## Best Practices
82+
83+
1. **Retrieve more results initially**: Fetch 100-200 candidates to give rerankers sufficient options to work with
84+
2. **Apply rerankers before RANGE**: Rerank first, then limit the number of results returned
85+
3. **Choose the right reranker**: Use RRF for combining searches, MMR for diversification
86+
4. **Test with your data**: Experiment with different parameters to find what works best for your use case
87+
88+
## Common Patterns
89+
90+
```rust
91+
// Pattern 1: Simple diversification
92+
SearchV<Document>(vec, 100)::RerankMMR(lambda: 0.7)::RANGE(0, 10)
93+
94+
// Pattern 2: Hybrid search fusion
95+
SearchV<Document>(vec, 100)::RerankRRF::RANGE(0, 10)
96+
97+
// Pattern 3: Fusion + diversification
98+
SearchV<Document>(vec, 150)::RerankRRF::RerankMMR(lambda: 0.6)::RANGE(0, 10)
99+
```

0 commit comments

Comments
 (0)