<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="labeau-allauzen-2018-learning">
    <titleInfo>
        <title>Learning with Noise-Contrastive Estimation: Easing training by learning to scale</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Matthieu</namePart>
        <namePart type="family">Labeau</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Alexandre</namePart>
        <namePart type="family">Allauzen</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2018-08</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 27th International Conference on Computational Linguistics</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Emily</namePart>
            <namePart type="given">M</namePart>
            <namePart type="family">Bender</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Leon</namePart>
            <namePart type="family">Derczynski</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Pierre</namePart>
            <namePart type="family">Isabelle</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Santa Fe, New Mexico, USA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Noise-Contrastive Estimation (NCE) is a learning criterion that is regularly used to train neural language models in place of Maximum Likelihood Estimation, since it avoids the computational bottleneck caused by the output softmax. In this paper, we analyse and explain some of the weaknesses of this objective function, linked to the mechanism of self-normalization, by closely monitoring comparative experiments. We then explore several remedies and modifications to propose tractable and efficient NCE training strategies. In particular, we propose to make the scaling factor a trainable parameter of the model, and to use the noise distribution to initialize the output bias. These solutions, yet simple, yield stable and competitive performances in either small and large scale language modelling tasks.</abstract>
    <identifier type="citekey">labeau-allauzen-2018-learning</identifier>
    <location>
        <url>https://aclanthology.org/C18-1261</url>
    </location>
    <part>
        <date>2018-08</date>
        <extent unit="page">
            <start>3090</start>
            <end>3101</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Learning with Noise-Contrastive Estimation: Easing training by learning to scale
%A Labeau, Matthieu
%A Allauzen, Alexandre
%Y Bender, Emily M.
%Y Derczynski, Leon
%Y Isabelle, Pierre
%S Proceedings of the 27th International Conference on Computational Linguistics
%D 2018
%8 August
%I Association for Computational Linguistics
%C Santa Fe, New Mexico, USA
%F labeau-allauzen-2018-learning
%X Noise-Contrastive Estimation (NCE) is a learning criterion that is regularly used to train neural language models in place of Maximum Likelihood Estimation, since it avoids the computational bottleneck caused by the output softmax. In this paper, we analyse and explain some of the weaknesses of this objective function, linked to the mechanism of self-normalization, by closely monitoring comparative experiments. We then explore several remedies and modifications to propose tractable and efficient NCE training strategies. In particular, we propose to make the scaling factor a trainable parameter of the model, and to use the noise distribution to initialize the output bias. These solutions, yet simple, yield stable and competitive performances in either small and large scale language modelling tasks.
%U https://aclanthology.org/C18-1261
%P 3090-3101

Markdown (Informal)

[Learning with Noise-Contrastive Estimation: Easing training by learning to scale](https://aclanthology.org/C18-1261) (Labeau & Allauzen, COLING 2018)

• Learning with Noise-Contrastive Estimation: Easing training by learning to scale (Labeau & Allauzen, COLING 2018)

ACL

• Matthieu Labeau and Alexandre Allauzen. 2018. Learning with Noise-Contrastive Estimation: Easing training by learning to scale. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3090–3101, Santa Fe, New Mexico, USA. Association for Computational Linguistics.