<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="byrd-srivastava-2022-predicting">
    <titleInfo>
        <title>Predicting Difficulty and Discrimination of Natural Language Questions</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Matthew</namePart>
        <namePart type="family">Byrd</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Shashank</namePart>
        <namePart type="family">Srivastava</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2022-05</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Smaranda</namePart>
            <namePart type="family">Muresan</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Preslav</namePart>
            <namePart type="family">Nakov</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Aline</namePart>
            <namePart type="family">Villavicencio</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Dublin, Ireland</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Item Response Theory (IRT) has been extensively used to numerically characterize question difficulty and discrimination for human subjects in domains including cognitive psychology and education (Primi et al., 2014; Downing, 2003). More recently, IRT has been used to similarly characterize item difficulty and discrimination for natural language models across various datasets (Lalor et al., 2019; Vania et al., 2021; Rodriguez et al., 2021). In this work, we explore predictive models for directly estimating and explaining these traits for natural language questions in a question-answering context. We use HotpotQA for illustration. Our experiments show that it is possible to predict both difficulty and discrimination parameters for new questions, and these traits are correlated with features of questions, answers, and associated contexts. Our findings can have significant implications for the creation of new datasets and tests on the one hand and strategies such as active learning and curriculum learning on the other.</abstract>
    <identifier type="citekey">byrd-srivastava-2022-predicting</identifier>
    <identifier type="doi">10.18653/v1/2022.acl-short.15</identifier>
    <location>
        <url>https://aclanthology.org/2022.acl-short.15</url>
    </location>
    <part>
        <date>2022-05</date>
        <extent unit="page">
            <start>119</start>
            <end>130</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Predicting Difficulty and Discrimination of Natural Language Questions
%A Byrd, Matthew
%A Srivastava, Shashank
%Y Muresan, Smaranda
%Y Nakov, Preslav
%Y Villavicencio, Aline
%S Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
%D 2022
%8 May
%I Association for Computational Linguistics
%C Dublin, Ireland
%F byrd-srivastava-2022-predicting
%X Item Response Theory (IRT) has been extensively used to numerically characterize question difficulty and discrimination for human subjects in domains including cognitive psychology and education (Primi et al., 2014; Downing, 2003). More recently, IRT has been used to similarly characterize item difficulty and discrimination for natural language models across various datasets (Lalor et al., 2019; Vania et al., 2021; Rodriguez et al., 2021). In this work, we explore predictive models for directly estimating and explaining these traits for natural language questions in a question-answering context. We use HotpotQA for illustration. Our experiments show that it is possible to predict both difficulty and discrimination parameters for new questions, and these traits are correlated with features of questions, answers, and associated contexts. Our findings can have significant implications for the creation of new datasets and tests on the one hand and strategies such as active learning and curriculum learning on the other.
%R 10.18653/v1/2022.acl-short.15
%U https://aclanthology.org/2022.acl-short.15
%U https://doi.org/10.18653/v1/2022.acl-short.15
%P 119-130

Markdown (Informal)

[Predicting Difficulty and Discrimination of Natural Language Questions](https://aclanthology.org/2022.acl-short.15) (Byrd & Srivastava, ACL 2022)

• Predicting Difficulty and Discrimination of Natural Language Questions (Byrd & Srivastava, ACL 2022)

ACL

• Matthew Byrd and Shashank Srivastava. 2022. Predicting Difficulty and Discrimination of Natural Language Questions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 119–130, Dublin, Ireland. Association for Computational Linguistics.