<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="yeh-etal-2021-comprehensive">
    <titleInfo>
        <title>A Comprehensive Assessment of Dialog Evaluation Metrics</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Yi-Ting</namePart>
        <namePart type="family">Yeh</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Maxine</namePart>
        <namePart type="family">Eskenazi</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Shikib</namePart>
        <namePart type="family">Mehri</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2021-11</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>The First Workshop on Evaluations and Assessments of Neural Conversation Systems</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Wei</namePart>
            <namePart type="family">Wei</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Bo</namePart>
            <namePart type="family">Dai</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Tuo</namePart>
            <namePart type="family">Zhao</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Lihong</namePart>
            <namePart type="family">Li</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Diyi</namePart>
            <namePart type="family">Yang</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Yun-Nung</namePart>
            <namePart type="family">Chen</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Y-Lan</namePart>
            <namePart type="family">Boureau</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Asli</namePart>
            <namePart type="family">Celikyilmaz</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Alborz</namePart>
            <namePart type="family">Geramifard</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Aman</namePart>
            <namePart type="family">Ahuja</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Haoming</namePart>
            <namePart type="family">Jiang</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Online</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Automatic evaluation metrics are a crucial component of dialog systems research. Standard language evaluation metrics are known to be ineffective for evaluating dialog. As such, recent research has proposed a number of novel, dialog-specific metrics that correlate better with human judgements. Due to the fast pace of research, many of these metrics have been assessed on different datasets and there has as yet been no time for a systematic comparison between them. To this end, this paper provides a comprehensive assessment of recently proposed dialog evaluation metrics on a number of datasets. In this paper, 23 different automatic evaluation metrics are evaluated on 10 different datasets. Furthermore, the metrics are assessed in different settings, to better qualify their respective strengths and weaknesses. This comprehensive assessment offers several takeaways pertaining to dialog evaluation metrics in general. It also suggests how to best assess evaluation metrics and indicates promising directions for future work.</abstract>
    <identifier type="citekey">yeh-etal-2021-comprehensive</identifier>
    <identifier type="doi">10.18653/v1/2021.eancs-1.3</identifier>
    <location>
        <url>https://aclanthology.org/2021.eancs-1.3</url>
    </location>
    <part>
        <date>2021-11</date>
        <extent unit="page">
            <start>15</start>
            <end>33</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T A Comprehensive Assessment of Dialog Evaluation Metrics
%A Yeh, Yi-Ting
%A Eskenazi, Maxine
%A Mehri, Shikib
%Y Wei, Wei
%Y Dai, Bo
%Y Zhao, Tuo
%Y Li, Lihong
%Y Yang, Diyi
%Y Chen, Yun-Nung
%Y Boureau, Y-Lan
%Y Celikyilmaz, Asli
%Y Geramifard, Alborz
%Y Ahuja, Aman
%Y Jiang, Haoming
%S The First Workshop on Evaluations and Assessments of Neural Conversation Systems
%D 2021
%8 November
%I Association for Computational Linguistics
%C Online
%F yeh-etal-2021-comprehensive
%X Automatic evaluation metrics are a crucial component of dialog systems research. Standard language evaluation metrics are known to be ineffective for evaluating dialog. As such, recent research has proposed a number of novel, dialog-specific metrics that correlate better with human judgements. Due to the fast pace of research, many of these metrics have been assessed on different datasets and there has as yet been no time for a systematic comparison between them. To this end, this paper provides a comprehensive assessment of recently proposed dialog evaluation metrics on a number of datasets. In this paper, 23 different automatic evaluation metrics are evaluated on 10 different datasets. Furthermore, the metrics are assessed in different settings, to better qualify their respective strengths and weaknesses. This comprehensive assessment offers several takeaways pertaining to dialog evaluation metrics in general. It also suggests how to best assess evaluation metrics and indicates promising directions for future work.
%R 10.18653/v1/2021.eancs-1.3
%U https://aclanthology.org/2021.eancs-1.3
%U https://doi.org/10.18653/v1/2021.eancs-1.3
%P 15-33

Markdown (Informal)

[A Comprehensive Assessment of Dialog Evaluation Metrics](https://aclanthology.org/2021.eancs-1.3) (Yeh et al., EANCS 2021)

• A Comprehensive Assessment of Dialog Evaluation Metrics (Yeh et al., EANCS 2021)

ACL

• Yi-Ting Yeh, Maxine Eskenazi, and Shikib Mehri. 2021. A Comprehensive Assessment of Dialog Evaluation Metrics. In The First Workshop on Evaluations and Assessments of Neural Conversation Systems, pages 15–33, Online. Association for Computational Linguistics.