<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="sun-2016-asynchronous">
    <titleInfo>
        <title>Asynchronous Parallel Learning for Neural Networks and Structured Models with Dense Features</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Xu</namePart>
        <namePart type="family">Sun</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2016-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Yuji</namePart>
            <namePart type="family">Matsumoto</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Rashmi</namePart>
            <namePart type="family">Prasad</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>The COLING 2016 Organizing Committee</publisher>
            <place>
                <placeTerm type="text">Osaka, Japan</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Existing asynchronous parallel learning methods are only for the sparse feature models, and they face new challenges for the dense feature models like neural networks (e.g., LSTM, RNN). The problem for dense features is that asynchronous parallel learning brings gradient errors derived from overwrite actions. We show that gradient errors are very common and inevitable. Nevertheless, our theoretical analysis shows that the learning process with gradient errors can still be convergent towards the optimum of objective functions for many practical applications. Thus, we propose a simple method AsynGrad for asynchronous parallel learning with gradient error. Base on various dense feature models (LSTM, dense-CRF) and various NLP tasks, experiments show that AsynGrad achieves substantial improvement on training speed, and without any loss on accuracy.</abstract>
    <identifier type="citekey">sun-2016-asynchronous</identifier>
    <location>
        <url>https://aclanthology.org/C16-1019</url>
    </location>
    <part>
        <date>2016-12</date>
        <extent unit="page">
            <start>192</start>
            <end>202</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Asynchronous Parallel Learning for Neural Networks and Structured Models with Dense Features
%A Sun, Xu
%Y Matsumoto, Yuji
%Y Prasad, Rashmi
%S Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers
%D 2016
%8 December
%I The COLING 2016 Organizing Committee
%C Osaka, Japan
%F sun-2016-asynchronous
%X Existing asynchronous parallel learning methods are only for the sparse feature models, and they face new challenges for the dense feature models like neural networks (e.g., LSTM, RNN). The problem for dense features is that asynchronous parallel learning brings gradient errors derived from overwrite actions. We show that gradient errors are very common and inevitable. Nevertheless, our theoretical analysis shows that the learning process with gradient errors can still be convergent towards the optimum of objective functions for many practical applications. Thus, we propose a simple method AsynGrad for asynchronous parallel learning with gradient error. Base on various dense feature models (LSTM, dense-CRF) and various NLP tasks, experiments show that AsynGrad achieves substantial improvement on training speed, and without any loss on accuracy.
%U https://aclanthology.org/C16-1019
%P 192-202

Markdown (Informal)

[Asynchronous Parallel Learning for Neural Networks and Structured Models with Dense Features](https://aclanthology.org/C16-1019) (Sun, COLING 2016)

• Asynchronous Parallel Learning for Neural Networks and Structured Models with Dense Features (Sun, COLING 2016)

ACL

• Xu Sun. 2016. Asynchronous Parallel Learning for Neural Networks and Structured Models with Dense Features. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 192–202, Osaka, Japan. The COLING 2016 Organizing Committee.