Ground truth response in the evaluation dataset

[antonpibm]

This is a really nice and interesting work.

But is there a reason not to add the ground truth final response to the dataset?
It seems like the current evaluation relies on the evaluating model to infer this information from the history, why not add it to the dataset for better answer evaluation?

Thank you.

[lucenzhong]

It's hard to evaluate the model's response with a rule-based method. For information seeking functions, even if we label the golden answer, we can only use llm to evaluate the consistency between the generated model response and the golden answer. So, we use llm to evaluate the completeness and correctness of the model's response.

[antonpibm]

I totally agree that the evaluation is practically impossible using rule-based method.
What I suggest is that it's easier for LLMaaJ to compare a response against golden answer than to compare a response to the full tool usage trajectory, would you agree?
And to achieve this a golden answer is required, do you by chance have it available?

Thanks for the quick response!

[lucenzhong]

I agree with you. Comparing a response against golden answer is a good point. But we did not label the golden responses.