<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="nedelchev-etal-2020-language">
    <titleInfo>
        <title>Language Model Transformers as Evaluators for Open-domain Dialogues</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Rostislav</namePart>
        <namePart type="family">Nedelchev</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Jens</namePart>
        <namePart type="family">Lehmann</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Ricardo</namePart>
        <namePart type="family">Usbeck</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2020-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 28th International Conference on Computational Linguistics</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Donia</namePart>
            <namePart type="family">Scott</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Nuria</namePart>
            <namePart type="family">Bel</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Chengqing</namePart>
            <namePart type="family">Zong</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>International Committee on Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Barcelona, Spain (Online)</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Computer-based systems for communication with humans are a cornerstone of AI research since the 1950s. So far, the most effective way to assess the quality of the dialogues produced by these systems is to use resource-intensive manual labor instead of automated means. In this work, we investigate whether language models (LM) based on transformer neural networks can indicate the quality of a conversation. In a general sense, language models are methods that learn to predict one or more words based on an already given context. Due to their unsupervised nature, they are candidates for efficient, automatic indication of dialogue quality. We demonstrate that human evaluators have a positive correlation between the output of the language models and scores. We also provide some insights into their behavior and inner-working in a conversational context.</abstract>
    <identifier type="citekey">nedelchev-etal-2020-language</identifier>
    <identifier type="doi">10.18653/v1/2020.coling-main.599</identifier>
    <location>
        <url>https://aclanthology.org/2020.coling-main.599</url>
    </location>
    <part>
        <date>2020-12</date>
        <extent unit="page">
            <start>6797</start>
            <end>6808</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Language Model Transformers as Evaluators for Open-domain Dialogues
%A Nedelchev, Rostislav
%A Lehmann, Jens
%A Usbeck, Ricardo
%Y Scott, Donia
%Y Bel, Nuria
%Y Zong, Chengqing
%S Proceedings of the 28th International Conference on Computational Linguistics
%D 2020
%8 December
%I International Committee on Computational Linguistics
%C Barcelona, Spain (Online)
%F nedelchev-etal-2020-language
%X Computer-based systems for communication with humans are a cornerstone of AI research since the 1950s. So far, the most effective way to assess the quality of the dialogues produced by these systems is to use resource-intensive manual labor instead of automated means. In this work, we investigate whether language models (LM) based on transformer neural networks can indicate the quality of a conversation. In a general sense, language models are methods that learn to predict one or more words based on an already given context. Due to their unsupervised nature, they are candidates for efficient, automatic indication of dialogue quality. We demonstrate that human evaluators have a positive correlation between the output of the language models and scores. We also provide some insights into their behavior and inner-working in a conversational context.
%R 10.18653/v1/2020.coling-main.599
%U https://aclanthology.org/2020.coling-main.599
%U https://doi.org/10.18653/v1/2020.coling-main.599
%P 6797-6808

Markdown (Informal)

[Language Model Transformers as Evaluators for Open-domain Dialogues](https://aclanthology.org/2020.coling-main.599) (Nedelchev et al., COLING 2020)

• Language Model Transformers as Evaluators for Open-domain Dialogues (Nedelchev et al., COLING 2020)

ACL

• Rostislav Nedelchev, Jens Lehmann, and Ricardo Usbeck. 2020. Language Model Transformers as Evaluators for Open-domain Dialogues. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6797–6808, Barcelona, Spain (Online). International Committee on Computational Linguistics.