The Impacts of Unanswerable Questions on the Robustness of Machine Reading Comprehension Models

Son Quoc Tran, Phong Nguyen-Thuan Do, Uyen Le, Matt Kretchmar


Abstract
Pretrained language models have achieved super-human performances on many Machine Reading Comprehension (MRC) benchmarks. Nevertheless, their relative inability to defend against adversarial attacks has spurred skepticism about their natural language understanding. In this paper, we ask whether training with unanswerable questions in SQuAD 2.0 can help improve the robustness of MRC models against adversarial attacks. To explore that question, we fine-tune three state-of-the-art language models on either SQuAD 1.1 or SQuAD 2.0 and then evaluate their robustness under adversarial attacks. Our experiments reveal that current models fine-tuned on SQuAD 2.0 do not initially appear to be any more robust than ones fine-tuned on SQuAD 1.1, yet they reveal a measure of hidden robustness that can be leveraged to realize actual performance gains. Furthermore, we find that robustness of models fine-tuned on SQuAD 2.0 extends on additional out-of-domain datasets. Finally, we introduce a new adversarial attack to reveal of SQuAD 2.0 that current MRC models are learning.
Anthology ID:
2023.eacl-main.113
Volume:
Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics
Month:
May
Year:
2023
Address:
Dubrovnik, Croatia
Editors:
Andreas Vlachos, Isabelle Augenstein
Venue:
EACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1543–1557
Language:
URL:
https://aclanthology.org/2023.eacl-main.113
DOI:
10.18653/v1/2023.eacl-main.113
Bibkey:
Cite (ACL):
Son Quoc Tran, Phong Nguyen-Thuan Do, Uyen Le, and Matt Kretchmar. 2023. The Impacts of Unanswerable Questions on the Robustness of Machine Reading Comprehension Models. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1543–1557, Dubrovnik, Croatia. Association for Computational Linguistics.
Cite (Informal):
The Impacts of Unanswerable Questions on the Robustness of Machine Reading Comprehension Models (Tran et al., EACL 2023)
Copy Citation:
PDF:
https://aclanthology.org/2023.eacl-main.113.pdf
Video:
 https://aclanthology.org/2023.eacl-main.113.mp4