<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="pang-etal-2020-towards">
    <titleInfo>
        <title>Towards Holistic and Automatic Evaluation of Open-Domain Dialogue Generation</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Bo</namePart>
        <namePart type="family">Pang</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Erik</namePart>
        <namePart type="family">Nijkamp</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Wenjuan</namePart>
        <namePart type="family">Han</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Linqi</namePart>
        <namePart type="family">Zhou</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Yixian</namePart>
        <namePart type="family">Liu</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Kewei</namePart>
        <namePart type="family">Tu</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2020-07</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Dan</namePart>
            <namePart type="family">Jurafsky</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Joyce</namePart>
            <namePart type="family">Chai</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Natalie</namePart>
            <namePart type="family">Schluter</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Joel</namePart>
            <namePart type="family">Tetreault</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Online</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Open-domain dialogue generation has gained increasing attention in Natural Language Processing. Its evaluation requires a holistic means. Human ratings are deemed as the gold standard. As human evaluation is inefficient and costly, an automated substitute is highly desirable. In this paper, we propose holistic evaluation metrics that capture different aspects of open-domain dialogues. Our metrics consist of (1) GPT-2 based context coherence between sentences in a dialogue, (2) GPT-2 based fluency in phrasing, (3) n-gram based diversity in responses to augmented queries, and (4) textual-entailment-inference based logical self-consistency. The empirical validity of our metrics is demonstrated by strong correlations with human judgments. We open source the code and relevant materials.</abstract>
    <identifier type="citekey">pang-etal-2020-towards</identifier>
    <identifier type="doi">10.18653/v1/2020.acl-main.333</identifier>
    <location>
        <url>https://aclanthology.org/2020.acl-main.333</url>
    </location>
    <part>
        <date>2020-07</date>
        <extent unit="page">
            <start>3619</start>
            <end>3629</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Towards Holistic and Automatic Evaluation of Open-Domain Dialogue Generation
%A Pang, Bo
%A Nijkamp, Erik
%A Han, Wenjuan
%A Zhou, Linqi
%A Liu, Yixian
%A Tu, Kewei
%Y Jurafsky, Dan
%Y Chai, Joyce
%Y Schluter, Natalie
%Y Tetreault, Joel
%S Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
%D 2020
%8 July
%I Association for Computational Linguistics
%C Online
%F pang-etal-2020-towards
%X Open-domain dialogue generation has gained increasing attention in Natural Language Processing. Its evaluation requires a holistic means. Human ratings are deemed as the gold standard. As human evaluation is inefficient and costly, an automated substitute is highly desirable. In this paper, we propose holistic evaluation metrics that capture different aspects of open-domain dialogues. Our metrics consist of (1) GPT-2 based context coherence between sentences in a dialogue, (2) GPT-2 based fluency in phrasing, (3) n-gram based diversity in responses to augmented queries, and (4) textual-entailment-inference based logical self-consistency. The empirical validity of our metrics is demonstrated by strong correlations with human judgments. We open source the code and relevant materials.
%R 10.18653/v1/2020.acl-main.333
%U https://aclanthology.org/2020.acl-main.333
%U https://doi.org/10.18653/v1/2020.acl-main.333
%P 3619-3629

Markdown (Informal)

[Towards Holistic and Automatic Evaluation of Open-Domain Dialogue Generation](https://aclanthology.org/2020.acl-main.333) (Pang et al., ACL 2020)

• Towards Holistic and Automatic Evaluation of Open-Domain Dialogue Generation (Pang et al., ACL 2020)

ACL

• Bo Pang, Erik Nijkamp, Wenjuan Han, Linqi Zhou, Yixian Liu, and Kewei Tu. 2020. Towards Holistic and Automatic Evaluation of Open-Domain Dialogue Generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3619–3629, Online. Association for Computational Linguistics.