What Makes Reading Comprehension Questions Easier?

Saku Sugawara, Kentaro Inui, Satoshi Sekine, Akiko Aizawa


Abstract
A challenge in creating a dataset for machine reading comprehension (MRC) is to collect questions that require a sophisticated understanding of language to answer beyond using superficial cues. In this work, we investigate what makes questions easier across recent 12 MRC datasets with three question styles (answer extraction, description, and multiple choice). We propose to employ simple heuristics to split each dataset into easy and hard subsets and examine the performance of two baseline models for each of the subsets. We then manually annotate questions sampled from each subset with both validity and requisite reasoning skills to investigate which skills explain the difference between easy and hard questions. From this study, we observed that (i) the baseline performances for the hard subsets remarkably degrade compared to those of entire datasets, (ii) hard questions require knowledge inference and multiple-sentence reasoning in comparison with easy questions, and (iii) multiple-choice questions tend to require a broader range of reasoning skills than answer extraction and description questions. These results suggest that one might overestimate recent advances in MRC.
Anthology ID:
D18-1453
Volume:
Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
Month:
October-November
Year:
2018
Address:
Brussels, Belgium
Editors:
Ellen Riloff, David Chiang, Julia Hockenmaier, Jun’ichi Tsujii
Venue:
EMNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
4208–4219
Language:
URL:
https://aclanthology.org/D18-1453
DOI:
10.18653/v1/D18-1453
Bibkey:
Cite (ACL):
Saku Sugawara, Kentaro Inui, Satoshi Sekine, and Akiko Aizawa. 2018. What Makes Reading Comprehension Questions Easier?. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4208–4219, Brussels, Belgium. Association for Computational Linguistics.
Cite (Informal):
What Makes Reading Comprehension Questions Easier? (Sugawara et al., EMNLP 2018)
Copy Citation:
PDF:
https://aclanthology.org/D18-1453.pdf
Video:
 https://aclanthology.org/D18-1453.mp4
Code
 Alab-NII/mrc-heuristics
Data
ARC (AI2 Reasoning Challenge)MCScriptMCTestMS MARCONewsQARACESQuADTriviaQAWho-did-What