Large-scale Cloze Test Dataset Created by Teachers

Qizhe Xie, Guokun Lai, Zihang Dai, Eduard Hovy


Abstract
Cloze tests are widely adopted in language exams to evaluate students’ language proficiency. In this paper, we propose the first large-scale human-created cloze test dataset CLOTH, containing questions used in middle-school and high-school language exams. With missing blanks carefully created by teachers and candidate choices purposely designed to be nuanced, CLOTH requires a deeper language understanding and a wider attention span than previously automatically-generated cloze datasets. We test the performance of dedicatedly designed baseline models including a language model trained on the One Billion Word Corpus and show humans outperform them by a significant margin. We investigate the source of the performance gap, trace model deficiencies to some distinct properties of CLOTH, and identify the limited ability of comprehending the long-term context to be the key bottleneck.
Anthology ID:
D18-1257
Volume:
Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
Month:
October-November
Year:
2018
Address:
Brussels, Belgium
Venue:
EMNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
2344–2356
Language:
URL:
https://aclanthology.org/D18-1257
DOI:
10.18653/v1/D18-1257
Bibkey:
Cite (ACL):
Qizhe Xie, Guokun Lai, Zihang Dai, and Eduard Hovy. 2018. Large-scale Cloze Test Dataset Created by Teachers. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2344–2356, Brussels, Belgium. Association for Computational Linguistics.
Cite (Informal):
Large-scale Cloze Test Dataset Created by Teachers (Xie et al., EMNLP 2018)
Copy Citation:
PDF:
https://aclanthology.org/D18-1257.pdf
Video:
 https://vimeo.com/305886563
Code
 qizhex/Large-scale-Cloze-Test-Dataset-Designed-by-Teachers +  additional community code
Data
CLOTHBookCorpusCBTLAMBADA