How to combine the three variants of selfcheckgpt in your paper?

[dhx20150812]

Hi, @potsawee , thanks for sharing your nice work.

There is something unclear in your paper. I wonder if you can give some explanations?

After we obtain the three score, how to combine them for the final score? The three scores have different scale:

• $\mathcal{S}_{BERT} \in [0.0, 1.0]$
• $\mathcal{S}_{QA} \in [0.0, 1.0]$
• $\mathcal{S}_{n-gram} \in [0.0, inf]$

I noticed that in section 5.4, you wrote "As a result, we consider SelfCheckGPT-Combination, which is a simple combination of the normalized scores of the three variants...."

So what is the combination strategy?

Thanks in advance! Looking forward to your reply.

[potsawee]

Hi @dhx20150812

Sorry for the late reply. You are right that the n-gram score is not bounded between [0.0, 1.0] like other variants. To obtain the combination results (e.g., AUC-PR and correlation), I simply re-scaled the n-gram scores to make them bounded in [0.0, 1.0] by subtracting each n-gram score by the minimum value then divided by max-min, e.g., $x' = \frac{x - x_{\text{min}}}{x_{\text{max}} - x_{\text{min}}}$, where $x$ is the original n-gram score and $x'$ is thre re-scaled score.

Then the combined score is the average of the scores of various variants.

Hope this answers your question

[dhx20150812]

Thanks,

I think your explanations are useful and solve my doubts. I'll give it a try.

[dhx20150812]

Hi, @potsawee

I found a suspected error in your REMEMD file. You wrote:

SelfCheck-MQAG: Score for each sentence where value is in [0.0, 1.0] and high value means non-factual

SelfCheck-BERTScore: Score for each sentence where value is in [0.0, 1.0] and high value means non-factual

SelfCheck-Ngram: Score at sentence- and document-level where value is in [0.0, +inf) and high value means non-factual

However, in your original paper, in the first paragraph of section 5, you wrote:

we design SelfCheckGPT $\mathcal{S}(i)$ such that $\mathcal{S}(i) \in [0.0, 1.0]$ and $\mathcal{S}(i) \rightarrow 1.0$ if the $i$-th sentence is hallucinated, and $\mathcal{S}(i) \rightarrow 0.0$ if it is grounded in valid information.

which is contradictory to your doc.

I think there is something wrong with your doc. It is okay that high score mean hallucination and low score leads to factuality. Maybe you can check the doc. Hope this message helpful.

Best regards.

[oroojlooy]

@potsawee,
How do you normalize it if all sentences have the same n-gram value, or if there is only one sentence?
Do you set them to all zero or all one?

Hi @dhx20150812

Sorry for the late reply. You are right that the n-gram score is not bounded between [0.0, 1.0] like other variants. To obtain the combination results (e.g., AUC-PR and correlation), I simply re-scaled the n-gram scores to make them bounded in [0.0, 1.0] by subtracting each n-gram score by the minimum value then divided by max-min, e.g., x′=x−xminxmax−xmin, where x is the original n-gram score and x′ is thre re-scaled score.

Then the combined score is the average of the scores of various variants.

Hope this answers your question

[potsawee]

Hi @oroojlooy

The short and honest answer is that we hadn't thought/come across this scenario. It's a good point - thanks for raising this problem.

I'd assign a score of 0.5 to all sentences if all of them have the same n-gram score. This should be consistent with the QA-method ($S_{QA} = \frac{\gamma_2^{N'_n}}{\gamma_1^{N'_m}+\gamma_2^{N'_n}}$) when all questions are filtered out ($N'_m = N'_n = 0$).

[potsawee]

Hi, @potsawee

I found a suspected error in your REMEMD file. You wrote:

SelfCheck-MQAG: Score for each sentence where value is in [0.0, 1.0] and high value means non-factual

SelfCheck-BERTScore: Score for each sentence where value is in [0.0, 1.0] and high value means non-factual

SelfCheck-Ngram: Score at sentence- and document-level where value is in [0.0, +inf) and high value means non-factual

However, in your original paper, in the first paragraph of section 5, you wrote:

we design SelfCheckGPT S(i) such that S(i)∈[0.0,1.0] and S(i)→1.0 if the i-th sentence is hallucinated, and S(i)→0.0 if it is grounded in valid information.

which is contradictory to your doc.

I think there is something wrong with your doc. It is okay that high score mean hallucination and low score leads to factuality. Maybe you can check the doc. Hope this message helpful.

Best regards.

Hi @dhx20150812 ,

Thank you for raising this problem. Yes, I just realised that there is indeed a discrepancy between the arxiv paper and README.

The current doc should be correct, i.e., selfcheck-ngram is not bounded in $[0.0, 1.0]$. I will have to think whether to revise the definition of selfcheck-ngram to make it bounded. I will keep this issue open until I fix this problem.

Thank you very much for feedback!