What Makes Reading Comprehension Questions Difficult?

Saku Sugawara, Nikita Nangia, Alex Warstadt, Samuel Bowman


Abstract
For a natural language understanding benchmark to be useful in research, it has to consist of examples that are diverse and difficult enough to discriminate among current and near-future state-of-the-art systems. However, we do not yet know how best to select text sources to collect a variety of challenging examples. In this study, we crowdsource multiple-choice reading comprehension questions for passages taken from seven qualitatively distinct sources, analyzing what attributes of passages contribute to the difficulty and question types of the collected examples. To our surprise, we find that passage source, length, and readability measures do not significantly affect question difficulty. Through our manual annotation of seven reasoning types, we observe several trends between passage sources and reasoning types, e.g., logical reasoning is more often required in questions written for technical passages. These results suggest that when creating a new benchmark dataset, selecting a diverse set of passages can help ensure a diverse range of question types, but that passage difficulty need not be a priority.
Anthology ID:
2022.acl-long.479
Volume:
Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Month:
May
Year:
2022
Address:
Dublin, Ireland
Editors:
Smaranda Muresan, Preslav Nakov, Aline Villavicencio
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
6951–6971
Language:
URL:
https://aclanthology.org/2022.acl-long.479
DOI:
10.18653/v1/2022.acl-long.479
Bibkey:
Cite (ACL):
Saku Sugawara, Nikita Nangia, Alex Warstadt, and Samuel Bowman. 2022. What Makes Reading Comprehension Questions Difficult?. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6951–6971, Dublin, Ireland. Association for Computational Linguistics.
Cite (Informal):
What Makes Reading Comprehension Questions Difficult? (Sugawara et al., ACL 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.acl-long.479.pdf
Code
 nii-cl/qa-text-source-comparison
Data
MCTestMultiNLIRACEReClor