<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="finch-etal-2023-leveraging">
    <titleInfo>
        <title>Leveraging Large Language Models for Automated Dialogue Analysis</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Sarah</namePart>
        <namePart type="given">E</namePart>
        <namePart type="family">Finch</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Ellie</namePart>
        <namePart type="given">S</namePart>
        <namePart type="family">Paek</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Jinho</namePart>
        <namePart type="given">D</namePart>
        <namePart type="family">Choi</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-09</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Svetlana</namePart>
            <namePart type="family">Stoyanchev</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Shafiq</namePart>
            <namePart type="family">Joty</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">David</namePart>
            <namePart type="family">Schlangen</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Ondrej</namePart>
            <namePart type="family">Dusek</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Casey</namePart>
            <namePart type="family">Kennington</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Malihe</namePart>
            <namePart type="family">Alikhani</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Prague, Czechia</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Developing high-performing dialogue systems benefits from the automatic identification of undesirable behaviors in system responses. However, detecting such behaviors remains challenging, as it draws on a breadth of general knowledge and understanding of conversational practices. Although recent research has focused on building specialized classifiers for detecting specific dialogue behaviors, the behavior coverage is still incomplete and there is a lack of testing on real-world human-bot interactions. This paper investigates the ability of a state-of-the-art large language model (LLM), ChatGPT-3.5, to perform dialogue behavior detection for nine categories in real human-bot dialogues. We aim to assess whether ChatGPT can match specialized models and approximate human performance, thereby reducing the cost of behavior detection tasks. Our findings reveal that neither specialized models nor ChatGPT have yet achieved satisfactory results for this task, falling short of human performance. Nevertheless, ChatGPT shows promising potential and often outperforms specialized detection models. We conclude with an in-depth examination of the prevalent shortcomings of ChatGPT, offering guidance for future research to enhance LLM capabilities.</abstract>
    <identifier type="citekey">finch-etal-2023-leveraging</identifier>
    <identifier type="doi">10.18653/v1/2023.sigdial-1.20</identifier>
    <location>
        <url>https://aclanthology.org/2023.sigdial-1.20</url>
    </location>
    <part>
        <date>2023-09</date>
        <extent unit="page">
            <start>202</start>
            <end>215</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Leveraging Large Language Models for Automated Dialogue Analysis
%A Finch, Sarah E.
%A Paek, Ellie S.
%A Choi, Jinho D.
%Y Stoyanchev, Svetlana
%Y Joty, Shafiq
%Y Schlangen, David
%Y Dusek, Ondrej
%Y Kennington, Casey
%Y Alikhani, Malihe
%S Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue
%D 2023
%8 September
%I Association for Computational Linguistics
%C Prague, Czechia
%F finch-etal-2023-leveraging
%X Developing high-performing dialogue systems benefits from the automatic identification of undesirable behaviors in system responses. However, detecting such behaviors remains challenging, as it draws on a breadth of general knowledge and understanding of conversational practices. Although recent research has focused on building specialized classifiers for detecting specific dialogue behaviors, the behavior coverage is still incomplete and there is a lack of testing on real-world human-bot interactions. This paper investigates the ability of a state-of-the-art large language model (LLM), ChatGPT-3.5, to perform dialogue behavior detection for nine categories in real human-bot dialogues. We aim to assess whether ChatGPT can match specialized models and approximate human performance, thereby reducing the cost of behavior detection tasks. Our findings reveal that neither specialized models nor ChatGPT have yet achieved satisfactory results for this task, falling short of human performance. Nevertheless, ChatGPT shows promising potential and often outperforms specialized detection models. We conclude with an in-depth examination of the prevalent shortcomings of ChatGPT, offering guidance for future research to enhance LLM capabilities.
%R 10.18653/v1/2023.sigdial-1.20
%U https://aclanthology.org/2023.sigdial-1.20
%U https://doi.org/10.18653/v1/2023.sigdial-1.20
%P 202-215

Markdown (Informal)

[Leveraging Large Language Models for Automated Dialogue Analysis](https://aclanthology.org/2023.sigdial-1.20) (Finch et al., SIGDIAL 2023)

• Leveraging Large Language Models for Automated Dialogue Analysis (Finch et al., SIGDIAL 2023)

ACL

• Sarah E. Finch, Ellie S. Paek, and Jinho D. Choi. 2023. Leveraging Large Language Models for Automated Dialogue Analysis. In Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 202–215, Prague, Czechia. Association for Computational Linguistics.