<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="niehues-pham-2019-modeling">
    <titleInfo>
        <title>Modeling Confidence in Sequence-to-Sequence Models</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Jan</namePart>
        <namePart type="family">Niehues</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Ngoc-Quan</namePart>
        <namePart type="family">Pham</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2019-oct–nov</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 12th International Conference on Natural Language Generation</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Kees</namePart>
            <namePart type="family">van Deemter</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Chenghua</namePart>
            <namePart type="family">Lin</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Hiroya</namePart>
            <namePart type="family">Takamura</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Tokyo, Japan</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Recently, significant improvements have been achieved in various natural language processing tasks using neural sequence-to-sequence models. While aiming for the best generation quality is important, ultimately it is also necessary to develop models that can assess the quality of their output. In this work, we propose to use the similarity between training and test conditions as a measure for models’ confidence. We investigate methods solely using the similarity as well as methods combining it with the posterior probability. While traditionally only target tokens are annotated with confidence measures, we also investigate methods to annotate source tokens with confidence. By learning an internal alignment model, we can significantly improve confidence projection over using state-of-the-art external alignment tools. We evaluate the proposed methods on downstream confidence estimation for machine translation (MT). We show improvements on segment-level confidence estimation as well as on confidence estimation for source tokens. In addition, we show that the same methods can also be applied to other tasks using sequence-to-sequence models. On the automatic speech recognition (ASR) task, we are able to find 60% of the errors by looking at 20% of the data.</abstract>
    <identifier type="citekey">niehues-pham-2019-modeling</identifier>
    <identifier type="doi">10.18653/v1/W19-8671</identifier>
    <location>
        <url>https://aclanthology.org/W19-8671</url>
    </location>
    <part>
        <date>2019-oct–nov</date>
        <extent unit="page">
            <start>575</start>
            <end>583</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Modeling Confidence in Sequence-to-Sequence Models
%A Niehues, Jan
%A Pham, Ngoc-Quan
%Y van Deemter, Kees
%Y Lin, Chenghua
%Y Takamura, Hiroya
%S Proceedings of the 12th International Conference on Natural Language Generation
%D 2019
%8 oct–nov
%I Association for Computational Linguistics
%C Tokyo, Japan
%F niehues-pham-2019-modeling
%X Recently, significant improvements have been achieved in various natural language processing tasks using neural sequence-to-sequence models. While aiming for the best generation quality is important, ultimately it is also necessary to develop models that can assess the quality of their output. In this work, we propose to use the similarity between training and test conditions as a measure for models’ confidence. We investigate methods solely using the similarity as well as methods combining it with the posterior probability. While traditionally only target tokens are annotated with confidence measures, we also investigate methods to annotate source tokens with confidence. By learning an internal alignment model, we can significantly improve confidence projection over using state-of-the-art external alignment tools. We evaluate the proposed methods on downstream confidence estimation for machine translation (MT). We show improvements on segment-level confidence estimation as well as on confidence estimation for source tokens. In addition, we show that the same methods can also be applied to other tasks using sequence-to-sequence models. On the automatic speech recognition (ASR) task, we are able to find 60% of the errors by looking at 20% of the data.
%R 10.18653/v1/W19-8671
%U https://aclanthology.org/W19-8671
%U https://doi.org/10.18653/v1/W19-8671
%P 575-583

Markdown (Informal)

[Modeling Confidence in Sequence-to-Sequence Models](https://aclanthology.org/W19-8671) (Niehues & Pham, INLG 2019)

• Modeling Confidence in Sequence-to-Sequence Models (Niehues & Pham, INLG 2019)

ACL

• Jan Niehues and Ngoc-Quan Pham. 2019. Modeling Confidence in Sequence-to-Sequence Models. In Proceedings of the 12th International Conference on Natural Language Generation, pages 575–583, Tokyo, Japan. Association for Computational Linguistics.