<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="mendonca-etal-2024-ecoh">
    <titleInfo>
        <title>ECoh: Turn-level Coherence Evaluation for Multilingual Dialogues</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">John</namePart>
        <namePart type="family">Mendonca</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Isabel</namePart>
        <namePart type="family">Trancoso</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Alon</namePart>
        <namePart type="family">Lavie</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024-09</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 25th Annual Meeting of the Special Interest Group on Discourse and Dialogue</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Tatsuya</namePart>
            <namePart type="family">Kawahara</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Vera</namePart>
            <namePart type="family">Demberg</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Stefan</namePart>
            <namePart type="family">Ultes</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Koji</namePart>
            <namePart type="family">Inoue</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Shikib</namePart>
            <namePart type="family">Mehri</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">David</namePart>
            <namePart type="family">Howcroft</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Kazunori</namePart>
            <namePart type="family">Komatani</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Kyoto, Japan</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Despite being heralded as the new standard for dialogue evaluation, the closed-source nature of GPT-4 poses challenges for the community. Motivated by the need for lightweight, open source, and multilingual dialogue evaluators, this paper introduces GenResCoh (Generated Responses targeting Coherence). GenResCoh is a novel LLM generated dataset comprising over 130k negative and positive responses and accompanying explanations seeded from XDailyDialog and XPersona covering English, French, German, Italian, and Chinese. Leveraging GenResCoh, we propose ECoh (Evaluation of Coherence), a family of evaluators trained to assess response coherence across multiple languages. Experimental results demonstrate that ECoh achieves multilingual detection capabilities superior to the teacher model (GPT-3.5-Turbo) on GenResCoh, despite being based on a much smaller architecture. Furthermore, the explanations provided by ECoh closely align in terms of quality with those generated by the teacher model.</abstract>
    <identifier type="citekey">mendonca-etal-2024-ecoh</identifier>
    <identifier type="doi">10.18653/v1/2024.sigdial-1.44</identifier>
    <location>
        <url>https://aclanthology.org/2024.sigdial-1.44</url>
    </location>
    <part>
        <date>2024-09</date>
        <extent unit="page">
            <start>516</start>
            <end>532</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T ECoh: Turn-level Coherence Evaluation for Multilingual Dialogues
%A Mendonca, John
%A Trancoso, Isabel
%A Lavie, Alon
%Y Kawahara, Tatsuya
%Y Demberg, Vera
%Y Ultes, Stefan
%Y Inoue, Koji
%Y Mehri, Shikib
%Y Howcroft, David
%Y Komatani, Kazunori
%S Proceedings of the 25th Annual Meeting of the Special Interest Group on Discourse and Dialogue
%D 2024
%8 September
%I Association for Computational Linguistics
%C Kyoto, Japan
%F mendonca-etal-2024-ecoh
%X Despite being heralded as the new standard for dialogue evaluation, the closed-source nature of GPT-4 poses challenges for the community. Motivated by the need for lightweight, open source, and multilingual dialogue evaluators, this paper introduces GenResCoh (Generated Responses targeting Coherence). GenResCoh is a novel LLM generated dataset comprising over 130k negative and positive responses and accompanying explanations seeded from XDailyDialog and XPersona covering English, French, German, Italian, and Chinese. Leveraging GenResCoh, we propose ECoh (Evaluation of Coherence), a family of evaluators trained to assess response coherence across multiple languages. Experimental results demonstrate that ECoh achieves multilingual detection capabilities superior to the teacher model (GPT-3.5-Turbo) on GenResCoh, despite being based on a much smaller architecture. Furthermore, the explanations provided by ECoh closely align in terms of quality with those generated by the teacher model.
%R 10.18653/v1/2024.sigdial-1.44
%U https://aclanthology.org/2024.sigdial-1.44
%U https://doi.org/10.18653/v1/2024.sigdial-1.44
%P 516-532

Markdown (Informal)

[ECoh: Turn-level Coherence Evaluation for Multilingual Dialogues](https://aclanthology.org/2024.sigdial-1.44) (Mendonca et al., SIGDIAL 2024)

• ECoh: Turn-level Coherence Evaluation for Multilingual Dialogues (Mendonca et al., SIGDIAL 2024)

ACL

• John Mendonca, Isabel Trancoso, and Alon Lavie. 2024. ECoh: Turn-level Coherence Evaluation for Multilingual Dialogues. In Proceedings of the 25th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 516–532, Kyoto, Japan. Association for Computational Linguistics.