He Thinks He Knows Better than the Doctors: BERT for Event Factuality Fails on Pragmatics

Nanjiang Jiang, Marie-Catherine de Marneffe


Abstract
We investigate how well BERT performs on predicting factuality in several existing English datasets, encompassing various linguistic constructions. Although BERT obtains a strong performance on most datasets, it does so by exploiting common surface patterns that correlate with certain factuality labels, and it fails on instances where pragmatic reasoning is necessary. Contrary to what the high performance suggests, we are still far from having a robust system for factuality prediction.
Anthology ID:
2021.tacl-1.64
Volume:
Transactions of the Association for Computational Linguistics, Volume 9
Month:
Year:
2021
Address:
Cambridge, MA
Editors:
Brian Roark, Ani Nenkova
Venue:
TACL
SIG:
Publisher:
MIT Press
Note:
Pages:
1081–1097
Language:
URL:
https://aclanthology.org/2021.tacl-1.64
DOI:
10.1162/tacl_a_00414
Bibkey:
Cite (ACL):
Nanjiang Jiang and Marie-Catherine de Marneffe. 2021. He Thinks He Knows Better than the Doctors: BERT for Event Factuality Fails on Pragmatics. Transactions of the Association for Computational Linguistics, 9:1081–1097.
Cite (Informal):
He Thinks He Knows Better than the Doctors: BERT for Event Factuality Fails on Pragmatics (Jiang & de Marneffe, TACL 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.tacl-1.64.pdf
Video:
 https://aclanthology.org/2021.tacl-1.64.mp4