Using Linguistic Features to Predict the Response Process Complexity Associated with Answering Clinical MCQs

Victoria Yaneva, Daniel Jurich, Le An Ha, Peter Baldwin


Abstract
This study examines the relationship between the linguistic characteristics of a test item and the complexity of the response process required to answer it correctly. Using data from a large-scale medical licensing exam, clustering methods identified items that were similar with respect to their relative difficulty and relative response-time intensiveness to create low response process complexity and high response process complexity item classes. Interpretable models were used to investigate the linguistic features that best differentiated between these classes from a descriptive and predictive framework. Results suggest that nuanced features such as the number of ambiguous medical terms help explain response process complexity beyond superficial item characteristics such as word count. Yet, although linguistic features carry signal relevant to response process complexity, the classification of individual items remains challenging.
Anthology ID:
2021.bea-1.23
Volume:
Proceedings of the 16th Workshop on Innovative Use of NLP for Building Educational Applications
Month:
April
Year:
2021
Address:
Online
Venue:
BEA
SIG:
SIGEDU
Publisher:
Association for Computational Linguistics
Note:
Pages:
223–232
Language:
URL:
https://aclanthology.org/2021.bea-1.23
DOI:
Bibkey:
Cite (ACL):
Victoria Yaneva, Daniel Jurich, Le An Ha, and Peter Baldwin. 2021. Using Linguistic Features to Predict the Response Process Complexity Associated with Answering Clinical MCQs. In Proceedings of the 16th Workshop on Innovative Use of NLP for Building Educational Applications, pages 223–232, Online. Association for Computational Linguistics.
Cite (Informal):
Using Linguistic Features to Predict the Response Process Complexity Associated with Answering Clinical MCQs (Yaneva et al., BEA 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.bea-1.23.pdf