<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="mehri-eskenazi-2020-unsupervised">
    <titleInfo>
        <title>Unsupervised Evaluation of Interactive Dialog with DialoGPT</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Shikib</namePart>
        <namePart type="family">Mehri</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Maxine</namePart>
        <namePart type="family">Eskenazi</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2020-07</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Olivier</namePart>
            <namePart type="family">Pietquin</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Smaranda</namePart>
            <namePart type="family">Muresan</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Vivian</namePart>
            <namePart type="family">Chen</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Casey</namePart>
            <namePart type="family">Kennington</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">David</namePart>
            <namePart type="family">Vandyke</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Nina</namePart>
            <namePart type="family">Dethlefs</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Koji</namePart>
            <namePart type="family">Inoue</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Erik</namePart>
            <namePart type="family">Ekstedt</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Stefan</namePart>
            <namePart type="family">Ultes</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">1st virtual meeting</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>It is important to define meaningful and interpretable automatic evaluation metrics for open-domain dialog research. Standard language generation metrics have been shown to be ineffective for dialog. This paper introduces the FED metric (fine-grained evaluation of dialog), an automatic evaluation metric which uses DialoGPT, without any fine-tuning or supervision. It also introduces the FED dataset which is constructed by annotating a set of human-system and human-human conversations with eighteen fine-grained dialog qualities. The FED metric (1) does not rely on a ground-truth response, (2) does not require training data and (3) measures fine-grained dialog qualities at both the turn and whole dialog levels. FED attains moderate to strong correlation with human judgement at both levels.</abstract>
    <identifier type="citekey">mehri-eskenazi-2020-unsupervised</identifier>
    <identifier type="doi">10.18653/v1/2020.sigdial-1.28</identifier>
    <location>
        <url>https://aclanthology.org/2020.sigdial-1.28</url>
    </location>
    <part>
        <date>2020-07</date>
        <extent unit="page">
            <start>225</start>
            <end>235</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Unsupervised Evaluation of Interactive Dialog with DialoGPT
%A Mehri, Shikib
%A Eskenazi, Maxine
%Y Pietquin, Olivier
%Y Muresan, Smaranda
%Y Chen, Vivian
%Y Kennington, Casey
%Y Vandyke, David
%Y Dethlefs, Nina
%Y Inoue, Koji
%Y Ekstedt, Erik
%Y Ultes, Stefan
%S Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue
%D 2020
%8 July
%I Association for Computational Linguistics
%C 1st virtual meeting
%F mehri-eskenazi-2020-unsupervised
%X It is important to define meaningful and interpretable automatic evaluation metrics for open-domain dialog research. Standard language generation metrics have been shown to be ineffective for dialog. This paper introduces the FED metric (fine-grained evaluation of dialog), an automatic evaluation metric which uses DialoGPT, without any fine-tuning or supervision. It also introduces the FED dataset which is constructed by annotating a set of human-system and human-human conversations with eighteen fine-grained dialog qualities. The FED metric (1) does not rely on a ground-truth response, (2) does not require training data and (3) measures fine-grained dialog qualities at both the turn and whole dialog levels. FED attains moderate to strong correlation with human judgement at both levels.
%R 10.18653/v1/2020.sigdial-1.28
%U https://aclanthology.org/2020.sigdial-1.28
%U https://doi.org/10.18653/v1/2020.sigdial-1.28
%P 225-235

Markdown (Informal)

[Unsupervised Evaluation of Interactive Dialog with DialoGPT](https://aclanthology.org/2020.sigdial-1.28) (Mehri & Eskenazi, SIGDIAL 2020)

• Unsupervised Evaluation of Interactive Dialog with DialoGPT (Mehri & Eskenazi, SIGDIAL 2020)

ACL

• Shikib Mehri and Maxine Eskenazi. 2020. Unsupervised Evaluation of Interactive Dialog with DialoGPT. In Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 225–235, 1st virtual meeting. Association for Computational Linguistics.