<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="ha-etal-2019-predicting">
    <titleInfo>
        <title>Predicting the Difficulty of Multiple Choice Questions in a High-stakes Medical Exam</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Le</namePart>
        <namePart type="given">An</namePart>
        <namePart type="family">Ha</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Victoria</namePart>
        <namePart type="family">Yaneva</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Peter</namePart>
        <namePart type="family">Baldwin</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Janet</namePart>
        <namePart type="family">Mee</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2019-08</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Helen</namePart>
            <namePart type="family">Yannakoudakis</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Ekaterina</namePart>
            <namePart type="family">Kochmar</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Claudia</namePart>
            <namePart type="family">Leacock</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Nitin</namePart>
            <namePart type="family">Madnani</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Ildikó</namePart>
            <namePart type="family">Pilán</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Torsten</namePart>
            <namePart type="family">Zesch</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Florence, Italy</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Predicting the construct-relevant difficulty of Multiple-Choice Questions (MCQs) has the potential to reduce cost while maintaining the quality of high-stakes exams. In this paper, we propose a method for estimating the difficulty of MCQs from a high-stakes medical exam, where all questions were deliberately written to a common reading level. To accomplish this, we extract a large number of linguistic features and embedding types, as well as features quantifying the difficulty of the items for an automatic question-answering system. The results show that the proposed approach outperforms various baselines with a statistically significant difference. Best results were achieved when using the full feature set, where embeddings had the highest predictive power, followed by linguistic features. An ablation study of the various types of linguistic features suggested that information from all levels of linguistic processing contributes to predicting item difficulty, with features related to semantic ambiguity and the psycholinguistic properties of words having a slightly higher importance. Owing to its generic nature, the presented approach has the potential to generalize over other exams containing MCQs.</abstract>
    <identifier type="citekey">ha-etal-2019-predicting</identifier>
    <identifier type="doi">10.18653/v1/W19-4402</identifier>
    <location>
        <url>https://aclanthology.org/W19-4402</url>
    </location>
    <part>
        <date>2019-08</date>
        <extent unit="page">
            <start>11</start>
            <end>20</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Predicting the Difficulty of Multiple Choice Questions in a High-stakes Medical Exam
%A Ha, Le An
%A Yaneva, Victoria
%A Baldwin, Peter
%A Mee, Janet
%Y Yannakoudakis, Helen
%Y Kochmar, Ekaterina
%Y Leacock, Claudia
%Y Madnani, Nitin
%Y Pilán, Ildikó
%Y Zesch, Torsten
%S Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications
%D 2019
%8 August
%I Association for Computational Linguistics
%C Florence, Italy
%F ha-etal-2019-predicting
%X Predicting the construct-relevant difficulty of Multiple-Choice Questions (MCQs) has the potential to reduce cost while maintaining the quality of high-stakes exams. In this paper, we propose a method for estimating the difficulty of MCQs from a high-stakes medical exam, where all questions were deliberately written to a common reading level. To accomplish this, we extract a large number of linguistic features and embedding types, as well as features quantifying the difficulty of the items for an automatic question-answering system. The results show that the proposed approach outperforms various baselines with a statistically significant difference. Best results were achieved when using the full feature set, where embeddings had the highest predictive power, followed by linguistic features. An ablation study of the various types of linguistic features suggested that information from all levels of linguistic processing contributes to predicting item difficulty, with features related to semantic ambiguity and the psycholinguistic properties of words having a slightly higher importance. Owing to its generic nature, the presented approach has the potential to generalize over other exams containing MCQs.
%R 10.18653/v1/W19-4402
%U https://aclanthology.org/W19-4402
%U https://doi.org/10.18653/v1/W19-4402
%P 11-20

Markdown (Informal)

[Predicting the Difficulty of Multiple Choice Questions in a High-stakes Medical Exam](https://aclanthology.org/W19-4402) (Ha et al., BEA 2019)

• Predicting the Difficulty of Multiple Choice Questions in a High-stakes Medical Exam (Ha et al., BEA 2019)

ACL

• Le An Ha, Victoria Yaneva, Peter Baldwin, and Janet Mee. 2019. Predicting the Difficulty of Multiple Choice Questions in a High-stakes Medical Exam. In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 11–20, Florence, Italy. Association for Computational Linguistics.