<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="aji-etal-2019-combining">
    <titleInfo>
        <title>Combining Global Sparse Gradients with Local Gradients in Distributed Neural Network Training</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Alham</namePart>
        <namePart type="given">Fikri</namePart>
        <namePart type="family">Aji</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Kenneth</namePart>
        <namePart type="family">Heafield</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Nikolay</namePart>
        <namePart type="family">Bogoychev</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2019-11</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Kentaro</namePart>
            <namePart type="family">Inui</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jing</namePart>
            <namePart type="family">Jiang</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Vincent</namePart>
            <namePart type="family">Ng</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Xiaojun</namePart>
            <namePart type="family">Wan</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Hong Kong, China</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>One way to reduce network traffic in multi-node data-parallel stochastic gradient descent is to only exchange the largest gradients. However, doing so damages the gradient and degrades the model‘s performance. Transformer models degrade dramatically while the impact on RNNs is smaller. We restore gradient quality by combining the compressed global gradient with the node‘s locally computed uncompressed gradient. Neural machine translation experiments show that Transformer convergence is restored while RNNs converge faster. With our method, training on 4 nodes converges up to 1.5x as fast as with uncompressed gradients and scales 3.5x relative to single-node training.</abstract>
    <identifier type="citekey">aji-etal-2019-combining</identifier>
    <identifier type="doi">10.18653/v1/D19-1373</identifier>
    <location>
        <url>https://aclanthology.org/D19-1373/</url>
    </location>
    <part>
        <date>2019-11</date>
        <extent unit="page">
            <start>3626</start>
            <end>3631</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Combining Global Sparse Gradients with Local Gradients in Distributed Neural Network Training
%A Aji, Alham Fikri
%A Heafield, Kenneth
%A Bogoychev, Nikolay
%Y Inui, Kentaro
%Y Jiang, Jing
%Y Ng, Vincent
%Y Wan, Xiaojun
%S Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
%D 2019
%8 November
%I Association for Computational Linguistics
%C Hong Kong, China
%F aji-etal-2019-combining
%X One way to reduce network traffic in multi-node data-parallel stochastic gradient descent is to only exchange the largest gradients. However, doing so damages the gradient and degrades the model‘s performance. Transformer models degrade dramatically while the impact on RNNs is smaller. We restore gradient quality by combining the compressed global gradient with the node‘s locally computed uncompressed gradient. Neural machine translation experiments show that Transformer convergence is restored while RNNs converge faster. With our method, training on 4 nodes converges up to 1.5x as fast as with uncompressed gradients and scales 3.5x relative to single-node training.
%R 10.18653/v1/D19-1373
%U https://aclanthology.org/D19-1373/
%U https://doi.org/10.18653/v1/D19-1373
%P 3626-3631

Markdown (Informal)

[Combining Global Sparse Gradients with Local Gradients in Distributed Neural Network Training](https://aclanthology.org/D19-1373/) (Aji et al., EMNLP-IJCNLP 2019)

• Combining Global Sparse Gradients with Local Gradients in Distributed Neural Network Training (Aji et al., EMNLP-IJCNLP 2019)

ACL

• Alham Fikri Aji, Kenneth Heafield, and Nikolay Bogoychev. 2019. Combining Global Sparse Gradients with Local Gradients in Distributed Neural Network Training. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3626–3631, Hong Kong, China. Association for Computational Linguistics.