<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="yang-etal-2018-adaptations">
    <titleInfo>
        <title>Adaptations of ROUGE and BLEU to Better Evaluate Machine Reading Comprehension Task</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">An</namePart>
        <namePart type="family">Yang</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Kai</namePart>
        <namePart type="family">Liu</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Jing</namePart>
        <namePart type="family">Liu</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Yajuan</namePart>
        <namePart type="family">Lyu</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Sujian</namePart>
        <namePart type="family">Li</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2018-07</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the Workshop on Machine Reading for Question Answering</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Eunsol</namePart>
            <namePart type="family">Choi</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Minjoon</namePart>
            <namePart type="family">Seo</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Danqi</namePart>
            <namePart type="family">Chen</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Robin</namePart>
            <namePart type="family">Jia</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jonathan</namePart>
            <namePart type="family">Berant</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Melbourne, Australia</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Current evaluation metrics to question answering based machine reading comprehension (MRC) systems generally focus on the lexical overlap between candidate and reference answers, such as ROUGE and BLEU. However, bias may appear when these metrics are used for specific question types, especially questions inquiring yes-no opinions and entity lists. In this paper, we make adaptations on the metrics to better correlate n-gram overlap with the human judgment for answers to these two question types. Statistical analysis proves the effectiveness of our approach. Our adaptations may provide positive guidance for the development of real-scene MRC systems.</abstract>
    <identifier type="citekey">yang-etal-2018-adaptations</identifier>
    <identifier type="doi">10.18653/v1/W18-2611</identifier>
    <location>
        <url>https://aclanthology.org/W18-2611</url>
    </location>
    <part>
        <date>2018-07</date>
        <extent unit="page">
            <start>98</start>
            <end>104</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Adaptations of ROUGE and BLEU to Better Evaluate Machine Reading Comprehension Task
%A Yang, An
%A Liu, Kai
%A Liu, Jing
%A Lyu, Yajuan
%A Li, Sujian
%Y Choi, Eunsol
%Y Seo, Minjoon
%Y Chen, Danqi
%Y Jia, Robin
%Y Berant, Jonathan
%S Proceedings of the Workshop on Machine Reading for Question Answering
%D 2018
%8 July
%I Association for Computational Linguistics
%C Melbourne, Australia
%F yang-etal-2018-adaptations
%X Current evaluation metrics to question answering based machine reading comprehension (MRC) systems generally focus on the lexical overlap between candidate and reference answers, such as ROUGE and BLEU. However, bias may appear when these metrics are used for specific question types, especially questions inquiring yes-no opinions and entity lists. In this paper, we make adaptations on the metrics to better correlate n-gram overlap with the human judgment for answers to these two question types. Statistical analysis proves the effectiveness of our approach. Our adaptations may provide positive guidance for the development of real-scene MRC systems.
%R 10.18653/v1/W18-2611
%U https://aclanthology.org/W18-2611
%U https://doi.org/10.18653/v1/W18-2611
%P 98-104

Markdown (Informal)

[Adaptations of ROUGE and BLEU to Better Evaluate Machine Reading Comprehension Task](https://aclanthology.org/W18-2611) (Yang et al., ACL 2018)

• Adaptations of ROUGE and BLEU to Better Evaluate Machine Reading Comprehension Task (Yang et al., ACL 2018)

ACL

• An Yang, Kai Liu, Jing Liu, Yajuan Lyu, and Sujian Li. 2018. Adaptations of ROUGE and BLEU to Better Evaluate Machine Reading Comprehension Task. In Proceedings of the Workshop on Machine Reading for Question Answering, pages 98–104, Melbourne, Australia. Association for Computational Linguistics.