<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="shea-yu-2023-building">
    <titleInfo>
        <title>Building Persona Consistent Dialogue Agents with Offline Reinforcement Learning</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Ryan</namePart>
        <namePart type="family">Shea</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Zhou</namePart>
        <namePart type="family">Yu</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Houda</namePart>
            <namePart type="family">Bouamor</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Juan</namePart>
            <namePart type="family">Pino</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Kalika</namePart>
            <namePart type="family">Bali</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Singapore</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Maintaining a consistent persona is a key quality for any open domain dialogue system. Current state-of-the-art systems do this by training agents with supervised learning or online reinforcement learning (RL). However, systems trained with supervised learning often lack consistency as they are never punished for uttering contradictions. Additional training with RL can alleviate some of these issues, however the training process is expensive. Instead, we propose an offline RL framework to improve the persona consistency of dialogue systems. Our framework allows us to combine the advantages of previous methods as we can inexpensively train our model on existing data as in supervised learning, while punishing and rewarding specific utterances as in RL. We also introduce a simple importance sampling method to reduce the variance of importance weights in offline RL training which we call Variance-Reducing MLE-Initialized (VaRMI) importance sampling. Our automatic and human evaluations show that our framework improves both the persona consistency and dialogue quality of a state-of-the-art social chatbot.</abstract>
    <identifier type="citekey">shea-yu-2023-building</identifier>
    <identifier type="doi">10.18653/v1/2023.emnlp-main.110</identifier>
    <location>
        <url>https://aclanthology.org/2023.emnlp-main.110</url>
    </location>
    <part>
        <date>2023-12</date>
        <extent unit="page">
            <start>1778</start>
            <end>1795</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Building Persona Consistent Dialogue Agents with Offline Reinforcement Learning
%A Shea, Ryan
%A Yu, Zhou
%Y Bouamor, Houda
%Y Pino, Juan
%Y Bali, Kalika
%S Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
%D 2023
%8 December
%I Association for Computational Linguistics
%C Singapore
%F shea-yu-2023-building
%X Maintaining a consistent persona is a key quality for any open domain dialogue system. Current state-of-the-art systems do this by training agents with supervised learning or online reinforcement learning (RL). However, systems trained with supervised learning often lack consistency as they are never punished for uttering contradictions. Additional training with RL can alleviate some of these issues, however the training process is expensive. Instead, we propose an offline RL framework to improve the persona consistency of dialogue systems. Our framework allows us to combine the advantages of previous methods as we can inexpensively train our model on existing data as in supervised learning, while punishing and rewarding specific utterances as in RL. We also introduce a simple importance sampling method to reduce the variance of importance weights in offline RL training which we call Variance-Reducing MLE-Initialized (VaRMI) importance sampling. Our automatic and human evaluations show that our framework improves both the persona consistency and dialogue quality of a state-of-the-art social chatbot.
%R 10.18653/v1/2023.emnlp-main.110
%U https://aclanthology.org/2023.emnlp-main.110
%U https://doi.org/10.18653/v1/2023.emnlp-main.110
%P 1778-1795

Markdown (Informal)

[Building Persona Consistent Dialogue Agents with Offline Reinforcement Learning](https://aclanthology.org/2023.emnlp-main.110) (Shea & Yu, EMNLP 2023)

• Building Persona Consistent Dialogue Agents with Offline Reinforcement Learning (Shea & Yu, EMNLP 2023)

ACL

• Ryan Shea and Zhou Yu. 2023. Building Persona Consistent Dialogue Agents with Offline Reinforcement Learning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1778–1795, Singapore. Association for Computational Linguistics.