Revealing Weaknesses of Vietnamese Language Models Through Unanswerable Questions in Machine Reading Comprehension

Son Quoc Tran, Phong Nguyen-Thuan Do, Kiet Van Nguyen, Ngan Luu-Thuy Nguyen


Abstract
Although the curse of multilinguality significantly restricts the language abilities of multilingual models in monolingual settings, researchers now still have to rely on multilingual models to develop state-of-the-art systems in Vietnamese Machine Reading Comprehension. This difficulty in researching is because of the limited number of high-quality works in developing Vietnamese language models. In order to encourage more work in this research field, we present a comprehensive analysis of language weaknesses and strengths of current Vietnamese monolingual models using the downstream task of Machine Reading Comprehension. From the analysis results, we suggest new directions for developing Vietnamese language models. Besides this main contribution, we also successfully reveal the existence of artifacts in Vietnamese Machine Reading Comprehension benchmarks and suggest an urgent need for new high-quality benchmarks to track the progress of Vietnamese Machine Reading Comprehension. Moreover, we also introduced a minor but valuable modification to the process of annotating unanswerable questions for Machine Reading Comprehension from previous work. Our proposed modification helps improve the quality of unanswerable questions to a higher level of difficulty for Machine Reading Comprehension systems to solve.
Anthology ID:
2023.eacl-srw.1
Volume:
Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop
Month:
May
Year:
2023
Address:
Dubrovnik, Croatia
Editors:
Elisa Bassignana, Matthias Lindemann, Alban Petit
Venue:
EACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1–13
Language:
URL:
https://aclanthology.org/2023.eacl-srw.1
DOI:
10.18653/v1/2023.eacl-srw.1
Bibkey:
Cite (ACL):
Son Quoc Tran, Phong Nguyen-Thuan Do, Kiet Van Nguyen, and Ngan Luu-Thuy Nguyen. 2023. Revealing Weaknesses of Vietnamese Language Models Through Unanswerable Questions in Machine Reading Comprehension. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 1–13, Dubrovnik, Croatia. Association for Computational Linguistics.
Cite (Informal):
Revealing Weaknesses of Vietnamese Language Models Through Unanswerable Questions in Machine Reading Comprehension (Tran et al., EACL 2023)
Copy Citation:
PDF:
https://aclanthology.org/2023.eacl-srw.1.pdf
Video:
 https://aclanthology.org/2023.eacl-srw.1.mp4