Bug: Intra-batch duplicate IDs create orphaned vectors (LangChain, Agno)
Files: langchain.py:170-208, agno.py:300-371
Description
When add_texts / add_documents / insert receives multiple documents with the same ID in a single batch, the second entry silently overwrites the mapping in _str_to_u64 while the first vector remains in the index. The first vector becomes an orphan — reachable by search but mapped to the wrong document text.
LlamaIndex correctly rejects this case (llama_index.py:140-153). Haystack handles it via DuplicatePolicy. LangChain and Agno have no guard.
Reproducer (LangChain)
from turbovec.langchain import TurboQuantVectorStore
from langchain_core.embeddings import Embeddings
class Stub(Embeddings):
def embed_documents(self, texts):
return [[float(i + 1)] * 64 for i in range(len(texts))]
def embed_query(self, t): return [1.0] * 64
store = TurboQuantVectorStore(Stub())
store.add_texts(["alpha", "beta"], ids=["dup", "dup"])
# Index has 2 vectors, but _str_to_u64 only maps "dup" → handle_2
assert len(store._index) == 2 # orphaned handle_1 exists
assert store._docs["dup"][0] == "beta" # "alpha" text is lostExpected behaviour
Reject intra-batch duplicate IDs with a ValueError, matching the LlamaIndex integration's behaviour (llama_index.py:140-153).
Impact
Silent data corruption — orphaned vectors pollute search results, and the "lost" document text is irrecoverable.
LlamaIndex handles this correctly at llama_index.py:140-153 by rejecting duplicates up front. Haystack covers it via DuplicatePolicy. But LangChain and Agno have no guard, so the second document in a batch silently orphans the first vector.
The fix would be adding the same intra-batch duplicate check that LlamaIndex already has. Fairly straightforward.
I'm working on building my open-source portfolio and would be happy to submit a fix. Would you be open to granting contributor access? If my work doesn't hit the mark, feel free to revoke it — no hard feelings.
Bug: Intra-batch duplicate IDs create orphaned vectors (LangChain, Agno)
Files:
langchain.py:170-208,agno.py:300-371Description
When
add_texts/add_documents/insertreceives multiple documents with the same ID in a single batch, the second entry silently overwrites the mapping in_str_to_u64while the first vector remains in the index. The first vector becomes an orphan — reachable by search but mapped to the wrong document text.LlamaIndex correctly rejects this case (
llama_index.py:140-153). Haystack handles it viaDuplicatePolicy. LangChain and Agno have no guard.Reproducer (LangChain)
Expected behaviour
Reject intra-batch duplicate IDs with a
ValueError, matching the LlamaIndex integration's behaviour (llama_index.py:140-153).Impact
Silent data corruption — orphaned vectors pollute search results, and the "lost" document text is irrecoverable.
LlamaIndex handles this correctly at
llama_index.py:140-153by rejecting duplicates up front. Haystack covers it viaDuplicatePolicy. But LangChain and Agno have no guard, so the second document in a batch silently orphans the first vector.The fix would be adding the same intra-batch duplicate check that LlamaIndex already has. Fairly straightforward.
I'm working on building my open-source portfolio and would be happy to submit a fix. Would you be open to granting contributor access? If my work doesn't hit the mark, feel free to revoke it — no hard feelings.