DialFact: A Benchmark for Fact-Checking in Dialogue

Prakhar Gupta, Chien-Sheng Wu, Wenhao Liu, Caiming Xiong


Abstract
Fact-checking is an essential tool to mitigate the spread of misinformation and disinformation. We introduce the task of fact-checking in dialogue, which is a relatively unexplored area. We construct DialFact, a testing benchmark dataset of 22,245 annotated conversational claims, paired with pieces of evidence from Wikipedia. There are three sub-tasks in DialFact: 1) Verifiable claim detection task distinguishes whether a response carries verifiable factual information; 2) Evidence retrieval task retrieves the most relevant Wikipedia snippets as evidence; 3) Claim verification task predicts a dialogue response to be supported, refuted, or not enough information. We found that existing fact-checking models trained on non-dialogue data like FEVER fail to perform well on our task, and thus, we propose a simple yet data-efficient solution to effectively improve fact-checking performance in dialogue. We point out unique challenges in DialFact such as handling the colloquialisms, coreferences, and retrieval ambiguities in the error analysis to shed light on future research in this direction.
Anthology ID:
2022.acl-long.263
Volume:
Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Month:
May
Year:
2022
Address:
Dublin, Ireland
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
3785–3801
Language:
URL:
https://aclanthology.org/2022.acl-long.263
DOI:
10.18653/v1/2022.acl-long.263
Bibkey:
Cite (ACL):
Prakhar Gupta, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2022. DialFact: A Benchmark for Fact-Checking in Dialogue. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3785–3801, Dublin, Ireland. Association for Computational Linguistics.
Cite (Informal):
DialFact: A Benchmark for Fact-Checking in Dialogue (Gupta et al., ACL 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.acl-long.263.pdf
Code
 salesforce/dialfact
Data
DialFactFEVERVitaminCWizard of Wikipedia