<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="mielke-etal-2022-reducing">
    <titleInfo>
        <title>Reducing Conversational Agents’ Overconfidence Through Linguistic Calibration</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Sabrina</namePart>
        <namePart type="given">J</namePart>
        <namePart type="family">Mielke</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Arthur</namePart>
        <namePart type="family">Szlam</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Emily</namePart>
        <namePart type="family">Dinan</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Y-Lan</namePart>
        <namePart type="family">Boureau</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2022</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <genre authority="bibutilsgt">journal article</genre>
    <relatedItem type="host">
        <titleInfo>
            <title>Transactions of the Association for Computational Linguistics</title>
        </titleInfo>
        <originInfo>
            <issuance>continuing</issuance>
            <publisher>MIT Press</publisher>
            <place>
                <placeTerm type="text">Cambridge, MA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">periodical</genre>
        <genre authority="bibutilsgt">academic journal</genre>
    </relatedItem>
    <abstract>While improving neural dialogue agents’ factual accuracy is the object of much research, another important aspect of communication, less studied in the setting of neural dialogue, is transparency about ignorance. In this work, we analyze to what extent state-of-the-art chit-chat models are linguistically calibrated in the sense that their verbalized expression of doubt (or confidence) matches the likelihood that the model’s responses are factually incorrect (or correct). We find that these models are poorly calibrated, yet we show that likelihood of correctness can accurately be predicted. By incorporating such metacognitive features into the training of a controllable generation model, we obtain a dialogue agent with greatly improved linguistic calibration.</abstract>
    <identifier type="citekey">mielke-etal-2022-reducing</identifier>
    <identifier type="doi">10.1162/tacl_a_00494</identifier>
    <location>
        <url>https://aclanthology.org/2022.tacl-1.50</url>
    </location>
    <part>
        <date>2022</date>
        <detail type="volume"><number>10</number></detail>
        <extent unit="page">
            <start>857</start>
            <end>872</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Journal Article
%T Reducing Conversational Agents’ Overconfidence Through Linguistic Calibration
%A Mielke, Sabrina J.
%A Szlam, Arthur
%A Dinan, Emily
%A Boureau, Y-Lan
%J Transactions of the Association for Computational Linguistics
%D 2022
%V 10
%I MIT Press
%C Cambridge, MA
%F mielke-etal-2022-reducing
%X While improving neural dialogue agents’ factual accuracy is the object of much research, another important aspect of communication, less studied in the setting of neural dialogue, is transparency about ignorance. In this work, we analyze to what extent state-of-the-art chit-chat models are linguistically calibrated in the sense that their verbalized expression of doubt (or confidence) matches the likelihood that the model’s responses are factually incorrect (or correct). We find that these models are poorly calibrated, yet we show that likelihood of correctness can accurately be predicted. By incorporating such metacognitive features into the training of a controllable generation model, we obtain a dialogue agent with greatly improved linguistic calibration.
%R 10.1162/tacl_a_00494
%U https://aclanthology.org/2022.tacl-1.50
%U https://doi.org/10.1162/tacl_a_00494
%P 857-872

Markdown (Informal)

[Reducing Conversational Agents’ Overconfidence Through Linguistic Calibration](https://aclanthology.org/2022.tacl-1.50) (Mielke et al., TACL 2022)

• Reducing Conversational Agents’ Overconfidence Through Linguistic Calibration (Mielke et al., TACL 2022)

ACL

• Sabrina J. Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau. 2022. Reducing Conversational Agents’ Overconfidence Through Linguistic Calibration. Transactions of the Association for Computational Linguistics, 10:857–872.