Nillion
Issue with Query Execution in NilDB - Internal DB Error (RepositoryError) #180
-
|
Hi, I'm working on a RAG POC following the examples provided in the nilrag repo. Under this schema I have posted some data that looks as follows: I have also posted a differences query, I'm using the examples provided as reference and adjusting it to fit my schema and use case. It looks as follows. Lastly I'm trying to execute this method. def diff_query_execute(self, nilql_query_embedding: list[list[bytes]], user_wallet_id: str, agent_id: str):
"""
Execute the difference query across all nilDB nodes.
Args:
nilql_query_embedding (list): Encrypted query embedding for all nilDB node.
Example:
[
# First element of embedding vector, shared across nodes
[
b'share_for_node_0', # Share for node 0
b'share_for_node_1', # Share for node 1
b'share_for_node_2' # Share for node 2
],
# Second element of embedding vector, shared across nodes
[
b'share_for_node_0',
b'share_for_node_1',
b'share_for_node_2'
],
# And so on for each element in the embedding vector...
]
Returns:
list: List of difference shares from each nilDB node
Raises:
ValueError: If query execution fails on any nilDB node
"""
# Rearrange nilql_query_embedding to group by party
query_embedding_shares = [
[entry[party] for entry in nilql_query_embedding]
for party in range(len(self.nodes))
]
#print(f"Query embedding shares: {query_embedding_shares}")
difference_shares = []
for i, node in enumerate(self.nodes):
url = node.url + "/queries/execute"
# Authorization header with the provided token
headers = {
"Authorization": "Bearer " + str(node.bearer_token),
"Content-Type": "application/json",
}
diff_query_id = node.diff_query_id
# Schema payload
payload = {
"id": str(diff_query_id),
"variables": {
"query_embedding": query_embedding_shares[i],
"userWalletId_filter": user_wallet_id, # Pass userWalletId_filter
"agent_id_filter": agent_id, # Pass agent_id_filter
},
}
print(f"Querying node {i} with payload: {payload}")
# Send POST request
response = requests.post(
url, headers=headers, data=json.dumps(payload), timeout=3600
)
if response.status_code != 200:
raise ValueError(
f"Error in POST request: {response.status_code}, {response.text}"
)
try:
difference_shares_party_i = response.json().get("data")
if difference_shares_party_i is None:
raise ValueError(f"Error in Response: {response.text}")
difference_shares.append(difference_shares_party_i)
except ValueError as e:
print(f"Failed to parse JSON response: {e}")
return response.text
return difference_sharesFrom the client I'm obtaining the query embeddings as follows additive_key = nilql.secret_key({'nodes': [{}] * 3}, {'sum': True})
# Generate query embedding from text
query_text = "what is the capital of France?"
query_embedding = generate_embeddings_huggingface([query_text])[0]
# Encrypt and share the query embedding
nilql_query_embedding = encrypt_float_list(additive_key, query_embedding)
wallet_id = '0x6d3F54E92Fd306199Fd90cD2e4E902E60Cf4F2dE'
agent_id = wallet_id
#print("Query embedding:", nilql_query_embedding)
# Execute query and get differences
difference_shares = nilDB.diff_query_execute(nilql_query_embedding, wallet_id, agent_id)With this my payload looks as follows: However, I'm getting the following error: My hypothesis is that the query embedding is not properly format in the diff query and or the payload but I have try with other formats without success so I'd like to ask if there's any feedback or debugging advice you can share to make this work. |
Beta Was this translation helpful? Give feedback.
Answered by
psofiterol
Mar 14, 2025
Replies: 2 comments 3 replies
-
|
Observation: In the as and it's also not present in the schema/collection. |
Beta Was this translation helpful? Give feedback.
3 replies
-
|
hey @psofiterol thanks a lot for the quick response. You're right. it seems I was missing the so I added it and now my query looks as follows: However, I'm still getting the same error message. I wonder if you have any tool or debug/log method that could bring more insights in this type of errors. Regarding your last comment where you mentioned Once again thanks for your support on this. |
Beta Was this translation helpful? Give feedback.
-
|
You're correct on schema_embeddings only being a variable 👌 - my comment about it not being on the schema was in reference to the pipeline not being able to find that value if it's not declared within it (and neither on the schema). It's clear that something on the query is still failing, but as is evident no detailed information on why it failed are being surfaced. The tool to use would probably be the query explanation tool in mongosh - I'll try that and see if that helps. |
Beta Was this translation helpful? Give feedback.
-
|
Alright, looks like it was basically one error in the pipeline definition -> the way
My suggestion is to swap this: For something closer to the example on the nilRAG library and not nest the embedings under |
Beta Was this translation helpful? Give feedback.
Answer selected by
emanuel-skai
-
|
Hey @psofiterol, thanks again for your support. Regarding the issue with $document_embeddings.$share not being supported in pipelines (since the $ is not supported on nested fields), this explanation makes sense. However, the schema I received included $document_embeddings.$share, which initially caused some confusion. I have now started from scratch by posting a new schema similar to the provided examples, and the differences query seems to be working fine. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Alright, looks like it was basically one error in the pipeline definition -> the way
$eqis used should be like this:$document_embeddings.$sharenot being supported in pipelines (the$not being supported on nested fields).My suggestion is to swap this: