Description:
The no_hallucination evaluation dimension is intermittently returning false in cases where the assistant's response appears to be correct and grounded.
Problem
In the linked trace, the user asks for help adding a date picker. The assistant adds the right things for implementing a date picker. However, the no_hallucination judge flags the response as hallucinated with the reasoning:
"The assistant's response includes multiple claims about adding a date picker and its requirements, none of which are supported by the provided context."
This suggests the judge is only checking whether claims appear verbatim in the retrieved documents, and treating any assistant-generated guidance (even when directly responding to the user's intent) as unsupported.
Suspected root cause
The MCP server indexes the documentation and the agent successfully retrieves relevant context — but the eval prompt likely only receives the retrieved context chunks as its "ground truth," not the full picture of what the agent had access to or what the user asked for.
This means:
The agent can reason correctly over the docs and produce a valid response
But the eval judge, with a narrower view, sees claims it can't trace back to a specific context chunk and flags them as hallucinations
Questions to investigate
What exactly is passed to the no_hallucination judge as context? Is it only the retrieved chunks, or does it also include the user query and conversation history?
Do we need to adjust the rubric so that the judge evaluates faithfulness to retrieved sources without penalizing legitimate responses to user queries that go beyond verbatim context?
Linked trace
Langfuse trace
Expected outcome
Clarify the evaluation contract for no_hallucination and update the judge prompt/rubric so that valid, user-requested responses are not incorrectly flagged as hallucinations.
Description:
The no_hallucination evaluation dimension is intermittently returning false in cases where the assistant's response appears to be correct and grounded.
Problem
In the linked trace, the user asks for help adding a date picker. The assistant adds the right things for implementing a date picker. However, the no_hallucination judge flags the response as hallucinated with the reasoning:
"The assistant's response includes multiple claims about adding a date picker and its requirements, none of which are supported by the provided context."
This suggests the judge is only checking whether claims appear verbatim in the retrieved documents, and treating any assistant-generated guidance (even when directly responding to the user's intent) as unsupported.
Suspected root cause
The MCP server indexes the documentation and the agent successfully retrieves relevant context — but the eval prompt likely only receives the retrieved context chunks as its "ground truth," not the full picture of what the agent had access to or what the user asked for.
This means:
The agent can reason correctly over the docs and produce a valid response
But the eval judge, with a narrower view, sees claims it can't trace back to a specific context chunk and flags them as hallucinations
Questions to investigate
What exactly is passed to the no_hallucination judge as context? Is it only the retrieved chunks, or does it also include the user query and conversation history?
Do we need to adjust the rubric so that the judge evaluates faithfulness to retrieved sources without penalizing legitimate responses to user queries that go beyond verbatim context?
Linked trace
Langfuse trace
Expected outcome
Clarify the evaluation contract for no_hallucination and update the judge prompt/rubric so that valid, user-requested responses are not incorrectly flagged as hallucinations.