<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="dabre-fujita-2021-investigating">
    <titleInfo>
        <title>Investigating Softmax Tempering for Training Neural Machine Translation Models</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Raj</namePart>
        <namePart type="family">Dabre</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Atsushi</namePart>
        <namePart type="family">Fujita</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2021-08</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of Machine Translation Summit XVIII: Research Track</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Kevin</namePart>
            <namePart type="family">Duh</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Francisco</namePart>
            <namePart type="family">Guzmán</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Machine Translation in the Americas</publisher>
            <place>
                <placeTerm type="text">Virtual</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Neural machine translation (NMT) models are typically trained using a softmax cross-entropy loss where the softmax distribution is compared against the gold labels. In low-resource scenarios and NMT models tend to perform poorly because the model training quickly converges to a point where the softmax distribution computed using logits approaches the gold label distribution. Although label smoothing is a well-known solution to address this issue and we further propose to divide the logits by a temperature coefficient greater than one and forcing the softmax distribution to be smoother during training. This makes it harder for the model to quickly over-fit. In our experiments on 11 language pairs in the low-resource Asian Language Treebank dataset and we observed significant improvements in translation quality. Our analysis focuses on finding the right balance of label smoothing and softmax tempering which indicates that they are orthogonal methods. Finally and a study of softmax entropies and gradients reveal the impact of our method on the internal behavior of our NMT models.</abstract>
    <identifier type="citekey">dabre-fujita-2021-investigating</identifier>
    <location>
        <url>https://aclanthology.org/2021.mtsummit-research.10</url>
    </location>
    <part>
        <date>2021-08</date>
        <extent unit="page">
            <start>114</start>
            <end>126</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Investigating Softmax Tempering for Training Neural Machine Translation Models
%A Dabre, Raj
%A Fujita, Atsushi
%Y Duh, Kevin
%Y Guzmán, Francisco
%S Proceedings of Machine Translation Summit XVIII: Research Track
%D 2021
%8 August
%I Association for Machine Translation in the Americas
%C Virtual
%F dabre-fujita-2021-investigating
%X Neural machine translation (NMT) models are typically trained using a softmax cross-entropy loss where the softmax distribution is compared against the gold labels. In low-resource scenarios and NMT models tend to perform poorly because the model training quickly converges to a point where the softmax distribution computed using logits approaches the gold label distribution. Although label smoothing is a well-known solution to address this issue and we further propose to divide the logits by a temperature coefficient greater than one and forcing the softmax distribution to be smoother during training. This makes it harder for the model to quickly over-fit. In our experiments on 11 language pairs in the low-resource Asian Language Treebank dataset and we observed significant improvements in translation quality. Our analysis focuses on finding the right balance of label smoothing and softmax tempering which indicates that they are orthogonal methods. Finally and a study of softmax entropies and gradients reveal the impact of our method on the internal behavior of our NMT models.
%U https://aclanthology.org/2021.mtsummit-research.10
%P 114-126

Markdown (Informal)

[Investigating Softmax Tempering for Training Neural Machine Translation Models](https://aclanthology.org/2021.mtsummit-research.10) (Dabre & Fujita, MTSummit 2021)

• Investigating Softmax Tempering for Training Neural Machine Translation Models (Dabre & Fujita, MTSummit 2021)

ACL

• Raj Dabre and Atsushi Fujita. 2021. Investigating Softmax Tempering for Training Neural Machine Translation Models. In Proceedings of Machine Translation Summit XVIII: Research Track, pages 114–126, Virtual. Association for Machine Translation in the Americas.