Predicting Item Survival for Multiple Choice Questions in a High-Stakes Medical Exam

Victoria Yaneva, Le An Ha, Peter Baldwin, Janet Mee


Abstract
One of the most resource-intensive problems in the educational testing industry relates to ensuring that newly-developed exam questions can adequately distinguish between students of high and low ability. The current practice for obtaining this information is the costly procedure of pretesting: new items are administered to test-takers and then the items that are too easy or too difficult are discarded. This paper presents the first study towards automatic prediction of an item’s probability to “survive” pretesting (item survival), focusing on human-produced MCQs for a medical exam. Survival is modeled through a number of linguistic features and embedding types, as well as features inspired by information retrieval. The approach shows promising first results for this challenging new application and for modeling the difficulty of expert-knowledge questions.
Anthology ID:
2020.lrec-1.841
Volume:
Proceedings of the Twelfth Language Resources and Evaluation Conference
Month:
May
Year:
2020
Address:
Marseille, France
Editors:
Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, Stelios Piperidis
Venue:
LREC
SIG:
Publisher:
European Language Resources Association
Note:
Pages:
6812–6818
Language:
English
URL:
https://aclanthology.org/2020.lrec-1.841
DOI:
Bibkey:
Cite (ACL):
Victoria Yaneva, Le An Ha, Peter Baldwin, and Janet Mee. 2020. Predicting Item Survival for Multiple Choice Questions in a High-Stakes Medical Exam. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 6812–6818, Marseille, France. European Language Resources Association.
Cite (Informal):
Predicting Item Survival for Multiple Choice Questions in a High-Stakes Medical Exam (Yaneva et al., LREC 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.lrec-1.841.pdf