REFIND: Retrieval Augmented Factual Hallucination Detection in Large Language Models

Task & Dataset Info.

SemEval-2025 Task-3 — Mu-SHROOM

Usage

Installation

conda create -n REFIND python=3.9
conda activate REFIND
pip install -r requirements.txt
python -m nltk.downloader punkt
python -m nltk.downloader punkt_tab

Preparation

Download Mu-SHROOM Dataset from Official Website and put it in the data directory.

# Retriever Preprocessing
sh scripts/preprocess_wiki.sh

Experiment

Validation Set

# Retrieve Contexts
sh scripts/run_val_retriever.sh

# Our Method
sh scripts/run_val_REFIND.sh

# Baselines
sh scripts/run_val_XLM-R.sh
sh scripts/run_val_FAVA.sh

## Evaluation
sh scripts/evaluate_val.sh

Test Set

# Retrieve Contexts
sh scripts/run_test_retriever.sh

# Our Method
sh scripts/run_test_REFIND.sh

# Baselines
sh scripts/run_test_XLM-R.sh
sh scripts/run_test_FAVA.sh

## Evaluation
sh scripts/evaluate_test.sh

References

• Official Website of Mu-SHROOM
• Official GitHub Repository of Mu-SHROOM
• Official Participant Kit of Mu-SHROOM