<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="inoue-etal-2020-r4c">
    <titleInfo>
        <title>R4C: A Benchmark for Evaluating RC Systems to Get the Right Answer for the Right Reason</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Naoya</namePart>
        <namePart type="family">Inoue</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Pontus</namePart>
        <namePart type="family">Stenetorp</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Kentaro</namePart>
        <namePart type="family">Inui</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2020-07</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Dan</namePart>
            <namePart type="family">Jurafsky</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Joyce</namePart>
            <namePart type="family">Chai</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Natalie</namePart>
            <namePart type="family">Schluter</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Joel</namePart>
            <namePart type="family">Tetreault</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Online</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Recent studies have revealed that reading comprehension (RC) systems learn to exploit annotation artifacts and other biases in current datasets. This prevents the community from reliably measuring the progress of RC systems. To address this issue, we introduce R4C, a new task for evaluating RC systems’ internal reasoning. R4C requires giving not only answers but also derivations: explanations that justify predicted answers. We present a reliable, crowdsourced framework for scalably annotating RC datasets with derivations. We create and publicly release the R4C dataset, the first, quality-assured dataset consisting of 4.6k questions, each of which is annotated with 3 reference derivations (i.e. 13.8k derivations). Experiments show that our automatic evaluation metrics using multiple reference derivations are reliable, and that R4C assesses different skills from an existing benchmark.</abstract>
    <identifier type="citekey">inoue-etal-2020-r4c</identifier>
    <identifier type="doi">10.18653/v1/2020.acl-main.602</identifier>
    <location>
        <url>https://aclanthology.org/2020.acl-main.602</url>
    </location>
    <part>
        <date>2020-07</date>
        <extent unit="page">
            <start>6740</start>
            <end>6750</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T R4C: A Benchmark for Evaluating RC Systems to Get the Right Answer for the Right Reason
%A Inoue, Naoya
%A Stenetorp, Pontus
%A Inui, Kentaro
%Y Jurafsky, Dan
%Y Chai, Joyce
%Y Schluter, Natalie
%Y Tetreault, Joel
%S Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
%D 2020
%8 July
%I Association for Computational Linguistics
%C Online
%F inoue-etal-2020-r4c
%X Recent studies have revealed that reading comprehension (RC) systems learn to exploit annotation artifacts and other biases in current datasets. This prevents the community from reliably measuring the progress of RC systems. To address this issue, we introduce R4C, a new task for evaluating RC systems’ internal reasoning. R4C requires giving not only answers but also derivations: explanations that justify predicted answers. We present a reliable, crowdsourced framework for scalably annotating RC datasets with derivations. We create and publicly release the R4C dataset, the first, quality-assured dataset consisting of 4.6k questions, each of which is annotated with 3 reference derivations (i.e. 13.8k derivations). Experiments show that our automatic evaluation metrics using multiple reference derivations are reliable, and that R4C assesses different skills from an existing benchmark.
%R 10.18653/v1/2020.acl-main.602
%U https://aclanthology.org/2020.acl-main.602
%U https://doi.org/10.18653/v1/2020.acl-main.602
%P 6740-6750

Markdown (Informal)

[R4C: A Benchmark for Evaluating RC Systems to Get the Right Answer for the Right Reason](https://aclanthology.org/2020.acl-main.602) (Inoue et al., ACL 2020)

• R4C: A Benchmark for Evaluating RC Systems to Get the Right Answer for the Right Reason (Inoue et al., ACL 2020)

ACL

• Naoya Inoue, Pontus Stenetorp, and Kentaro Inui. 2020. R4C: A Benchmark for Evaluating RC Systems to Get the Right Answer for the Right Reason. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6740–6750, Online. Association for Computational Linguistics.