<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="li-etal-2019-efficient">
    <titleInfo>
        <title>Efficient Contextual Representation Learning With Continuous Outputs</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Liunian</namePart>
        <namePart type="given">Harold</namePart>
        <namePart type="family">Li</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Patrick</namePart>
        <namePart type="given">H</namePart>
        <namePart type="family">Chen</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Cho-Jui</namePart>
        <namePart type="family">Hsieh</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Kai-Wei</namePart>
        <namePart type="family">Chang</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2019</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <genre authority="bibutilsgt">journal article</genre>
    <relatedItem type="host">
        <titleInfo>
            <title>Transactions of the Association for Computational Linguistics</title>
        </titleInfo>
        <originInfo>
            <issuance>continuing</issuance>
            <publisher>MIT Press</publisher>
            <place>
                <placeTerm type="text">Cambridge, MA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">periodical</genre>
        <genre authority="bibutilsgt">academic journal</genre>
    </relatedItem>
    <abstract>Contextual representation models have achieved great success in improving various downstream natural language processing tasks. However, these language-model-based encoders are difficult to train due to their large parameter size and high computational complexity. By carefully examining the training procedure, we observe that the softmax layer, which predicts a distribution of the target word, often induces significant overhead, especially when the vocabulary size is large. Therefore, we revisit the design of the output layer and consider directly predicting the pre-trained embedding of the target word for a given context. When applied to ELMo, the proposed approach achieves a 4-fold speedup and eliminates 80% trainable parameters while achieving competitive performance on downstream tasks. Further analysis shows that the approach maintains the speed advantage under various settings, even when the sentence encoder is scaled up.</abstract>
    <identifier type="citekey">li-etal-2019-efficient</identifier>
    <identifier type="doi">10.1162/tacl_a_00289</identifier>
    <location>
        <url>https://aclanthology.org/Q19-1039</url>
    </location>
    <part>
        <date>2019</date>
        <detail type="volume"><number>7</number></detail>
        <extent unit="page">
            <start>611</start>
            <end>624</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Journal Article
%T Efficient Contextual Representation Learning With Continuous Outputs
%A Li, Liunian Harold
%A Chen, Patrick H.
%A Hsieh, Cho-Jui
%A Chang, Kai-Wei
%J Transactions of the Association for Computational Linguistics
%D 2019
%V 7
%I MIT Press
%C Cambridge, MA
%F li-etal-2019-efficient
%X Contextual representation models have achieved great success in improving various downstream natural language processing tasks. However, these language-model-based encoders are difficult to train due to their large parameter size and high computational complexity. By carefully examining the training procedure, we observe that the softmax layer, which predicts a distribution of the target word, often induces significant overhead, especially when the vocabulary size is large. Therefore, we revisit the design of the output layer and consider directly predicting the pre-trained embedding of the target word for a given context. When applied to ELMo, the proposed approach achieves a 4-fold speedup and eliminates 80% trainable parameters while achieving competitive performance on downstream tasks. Further analysis shows that the approach maintains the speed advantage under various settings, even when the sentence encoder is scaled up.
%R 10.1162/tacl_a_00289
%U https://aclanthology.org/Q19-1039
%U https://doi.org/10.1162/tacl_a_00289
%P 611-624

Markdown (Informal)

[Efficient Contextual Representation Learning With Continuous Outputs](https://aclanthology.org/Q19-1039) (Li et al., TACL 2019)

• Efficient Contextual Representation Learning With Continuous Outputs (Li et al., TACL 2019)

ACL

• Liunian Harold Li, Patrick H. Chen, Cho-Jui Hsieh, and Kai-Wei Chang. 2019. Efficient Contextual Representation Learning With Continuous Outputs. Transactions of the Association for Computational Linguistics, 7:611–624.