How Robust are Fact Checking Systems on Colloquial Claims?

Byeongchang Kim, Hyunwoo Kim, Seokhee Hong, Gunhee Kim


Abstract
Knowledge is now starting to power neural dialogue agents. At the same time, the risk of misinformation and disinformation from dialogue agents also rises. Verifying the veracity of information from formal sources are widely studied in computational fact checking. In this work, we ask: How robust are fact checking systems on claims in colloquial style? We aim to open up new discussions in the intersection of fact verification and dialogue safety. In order to investigate how fact checking systems behave on colloquial claims, we transfer the styles of claims from FEVER (Thorne et al., 2018) into colloquialism. We find that existing fact checking systems that perform well on claims in formal style significantly degenerate on colloquial claims with the same semantics. Especially, we show that document retrieval is the weakest spot in the system even vulnerable to filler words, such as “yeah” and “you know”. The document recall of WikiAPI retriever (Hanselowski et al., 2018) which is 90.0% on FEVER, drops to 72.2% on the colloquial claims. We compare the characteristics of colloquial claims to those of claims in formal style, and demonstrate the challenging issues in them.
Anthology ID:
2021.naacl-main.121
Volume:
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Month:
June
Year:
2021
Address:
Online
Editors:
Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, Yichao Zhou
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1535–1548
Language:
URL:
https://aclanthology.org/2021.naacl-main.121
DOI:
10.18653/v1/2021.naacl-main.121
Bibkey:
Cite (ACL):
Byeongchang Kim, Hyunwoo Kim, Seokhee Hong, and Gunhee Kim. 2021. How Robust are Fact Checking Systems on Colloquial Claims?. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1535–1548, Online. Association for Computational Linguistics.
Cite (Informal):
How Robust are Fact Checking Systems on Colloquial Claims? (Kim et al., NAACL 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.naacl-main.121.pdf
Video:
 https://aclanthology.org/2021.naacl-main.121.mp4
Data
FEVERMultiNLIWizard of Wikipedia