Robust Machine Reading Comprehension by Learning Soft labels

Zhenyu Zhao, Shuangzhi Wu, Muyun Yang, Kehai Chen, Tiejun Zhao


Abstract
Neural models have achieved great success on the task of machine reading comprehension (MRC), which are typically trained on hard labels. We argue that hard labels limit the model capability on generalization due to the label sparseness problem. In this paper, we propose a robust training method for MRC models to address this problem. Our method consists of three strategies, 1) label smoothing, 2) word overlapping, 3) distribution prediction. All of them help to train models on soft labels. We validate our approach on the representative architecture - ALBERT. Experimental results show that our method can greatly boost the baseline with 1% improvement in average, and achieve state-of-the-art performance on NewsQA and QUOREF.
Anthology ID:
2020.coling-main.248
Volume:
Proceedings of the 28th International Conference on Computational Linguistics
Month:
December
Year:
2020
Address:
Barcelona, Spain (Online)
Editors:
Donia Scott, Nuria Bel, Chengqing Zong
Venue:
COLING
SIG:
Publisher:
International Committee on Computational Linguistics
Note:
Pages:
2754–2759
Language:
URL:
https://aclanthology.org/2020.coling-main.248
DOI:
10.18653/v1/2020.coling-main.248
Bibkey:
Cite (ACL):
Zhenyu Zhao, Shuangzhi Wu, Muyun Yang, Kehai Chen, and Tiejun Zhao. 2020. Robust Machine Reading Comprehension by Learning Soft labels. In Proceedings of the 28th International Conference on Computational Linguistics, pages 2754–2759, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Cite (Informal):
Robust Machine Reading Comprehension by Learning Soft labels (Zhao et al., COLING 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.coling-main.248.pdf
Data
NewsQAQuoref