<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="furniturewala-etal-2024-impact">
    <titleInfo>
        <title>Impact of Decoding Methods on Human Alignment of Conversational LLMs</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Shaz</namePart>
        <namePart type="family">Furniturewala</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Kokil</namePart>
        <namePart type="family">Jaidka</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Yashvardhan</namePart>
        <namePart type="family">Sharma</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024-08</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 14th Workshop on Computational Approaches to Subjectivity, Sentiment, &amp; Social Media Analysis</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Orphée</namePart>
            <namePart type="family">De Clercq</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Valentin</namePart>
            <namePart type="family">Barriere</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jeremy</namePart>
            <namePart type="family">Barnes</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Roman</namePart>
            <namePart type="family">Klinger</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">João</namePart>
            <namePart type="family">Sedoc</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Shabnam</namePart>
            <namePart type="family">Tafreshi</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Bangkok, Thailand</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>To be included into chatbot systems, Large language models (LLMs) must be aligned with human conversational conventions. However, being trained mainly on web-scraped data gives existing LLMs a voice closer to informational text than actual human speech. In this paper, we examine the effect of decoding methods on the alignment between LLM-generated and human conversations, including Beam Search, Top K Sampling, and Nucleus Sampling. We present new measures of alignment in substance, style, and psychometric orientation, and experiment with two conversation datasets. Our results provide subtle insights: better alignment is attributed to fewer beams in Beam Search and lower values of P in Nucleus Sampling. We also find that task-oriented and open-ended datasets perform differently in terms of alignment, indicating the significance of taking into account the context of the interaction.</abstract>
    <identifier type="citekey">furniturewala-etal-2024-impact</identifier>
    <identifier type="doi">10.18653/v1/2024.wassa-1.22</identifier>
    <location>
        <url>https://aclanthology.org/2024.wassa-1.22/</url>
    </location>
    <part>
        <date>2024-08</date>
        <extent unit="page">
            <start>273</start>
            <end>279</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Impact of Decoding Methods on Human Alignment of Conversational LLMs
%A Furniturewala, Shaz
%A Jaidka, Kokil
%A Sharma, Yashvardhan
%Y De Clercq, Orphée
%Y Barriere, Valentin
%Y Barnes, Jeremy
%Y Klinger, Roman
%Y Sedoc, João
%Y Tafreshi, Shabnam
%S Proceedings of the 14th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis
%D 2024
%8 August
%I Association for Computational Linguistics
%C Bangkok, Thailand
%F furniturewala-etal-2024-impact
%X To be included into chatbot systems, Large language models (LLMs) must be aligned with human conversational conventions. However, being trained mainly on web-scraped data gives existing LLMs a voice closer to informational text than actual human speech. In this paper, we examine the effect of decoding methods on the alignment between LLM-generated and human conversations, including Beam Search, Top K Sampling, and Nucleus Sampling. We present new measures of alignment in substance, style, and psychometric orientation, and experiment with two conversation datasets. Our results provide subtle insights: better alignment is attributed to fewer beams in Beam Search and lower values of P in Nucleus Sampling. We also find that task-oriented and open-ended datasets perform differently in terms of alignment, indicating the significance of taking into account the context of the interaction.
%R 10.18653/v1/2024.wassa-1.22
%U https://aclanthology.org/2024.wassa-1.22/
%U https://doi.org/10.18653/v1/2024.wassa-1.22
%P 273-279

Markdown (Informal)

[Impact of Decoding Methods on Human Alignment of Conversational LLMs](https://aclanthology.org/2024.wassa-1.22/) (Furniturewala et al., WASSA 2024)

• Impact of Decoding Methods on Human Alignment of Conversational LLMs (Furniturewala et al., WASSA 2024)

ACL

• Shaz Furniturewala, Kokil Jaidka, and Yashvardhan Sharma. 2024. Impact of Decoding Methods on Human Alignment of Conversational LLMs. In Proceedings of the 14th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis, pages 273–279, Bangkok, Thailand. Association for Computational Linguistics.