non-factual*

[coderpracticer]

thanks for your awesome work, i have a question. whether created dataset(1908 sentences) include the non-fatual* part?

[potsawee]

Hi @coderpracticer thanks for checking out the work.

1. Yes, the dataset [a] has all 3 labels (major_inaccurate, minor_inaccurate, accurate) at the sentence level.
2. Note that Non-Factual* denotes "less-trivial hallucination detection" as in our setup/definition, the percentage of hallucination is quite high (in some examples, all sentences are labelled as major-hallucination), we would like to see the performance of these methods in detecting more challenging hallucination detection. You can see in this notebook about how we created and used "Non-Factual*" labels (human_label_detect_False_h): https://github.com/potsawee/selfcheckgpt/blob/main/demo/experiments/probability-based-baselines.ipynb

Hope this clarifies your question

[a] Available at https://huggingface.co/datasets/potsawee/wiki_bio_gpt3_hallucination