<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="elbayad-etal-2018-token">
    <titleInfo>
        <title>Token-level and sequence-level loss smoothing for RNN language models</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Maha</namePart>
        <namePart type="family">Elbayad</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Laurent</namePart>
        <namePart type="family">Besacier</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Jakob</namePart>
        <namePart type="family">Verbeek</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2018-07</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Iryna</namePart>
            <namePart type="family">Gurevych</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Yusuke</namePart>
            <namePart type="family">Miyao</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Melbourne, Australia</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Despite the effectiveness of recurrent neural network language models, their maximum likelihood estimation suffers from two limitations. It treats all sentences that do not match the ground truth as equally poor, ignoring the structure of the output space. Second, it suffers from ’exposure bias’: during training tokens are predicted given ground-truth sequences, while at test time prediction is conditioned on generated output sequences. To overcome these limitations we build upon the recent reward augmented maximum likelihood approach that encourages the model to predict sentences that are close to the ground truth according to a given performance metric. We extend this approach to token-level loss smoothing, and propose improvements to the sequence-level smoothing approach. Our experiments on two different tasks, image captioning and machine translation, show that token-level and sequence-level loss smoothing are complementary, and significantly improve results.</abstract>
    <identifier type="citekey">elbayad-etal-2018-token</identifier>
    <identifier type="doi">10.18653/v1/P18-1195</identifier>
    <location>
        <url>https://aclanthology.org/P18-1195</url>
    </location>
    <part>
        <date>2018-07</date>
        <extent unit="page">
            <start>2094</start>
            <end>2103</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Token-level and sequence-level loss smoothing for RNN language models
%A Elbayad, Maha
%A Besacier, Laurent
%A Verbeek, Jakob
%Y Gurevych, Iryna
%Y Miyao, Yusuke
%S Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
%D 2018
%8 July
%I Association for Computational Linguistics
%C Melbourne, Australia
%F elbayad-etal-2018-token
%X Despite the effectiveness of recurrent neural network language models, their maximum likelihood estimation suffers from two limitations. It treats all sentences that do not match the ground truth as equally poor, ignoring the structure of the output space. Second, it suffers from ’exposure bias’: during training tokens are predicted given ground-truth sequences, while at test time prediction is conditioned on generated output sequences. To overcome these limitations we build upon the recent reward augmented maximum likelihood approach that encourages the model to predict sentences that are close to the ground truth according to a given performance metric. We extend this approach to token-level loss smoothing, and propose improvements to the sequence-level smoothing approach. Our experiments on two different tasks, image captioning and machine translation, show that token-level and sequence-level loss smoothing are complementary, and significantly improve results.
%R 10.18653/v1/P18-1195
%U https://aclanthology.org/P18-1195
%U https://doi.org/10.18653/v1/P18-1195
%P 2094-2103

Markdown (Informal)

[Token-level and sequence-level loss smoothing for RNN language models](https://aclanthology.org/P18-1195) (Elbayad et al., ACL 2018)

• Token-level and sequence-level loss smoothing for RNN language models (Elbayad et al., ACL 2018)

ACL

• Maha Elbayad, Laurent Besacier, and Jakob Verbeek. 2018. Token-level and sequence-level loss smoothing for RNN language models. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2094–2103, Melbourne, Australia. Association for Computational Linguistics.