<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="britz-etal-2017-efficient">
    <titleInfo>
        <title>Efficient Attention using a Fixed-Size Memory Representation</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Denny</namePart>
        <namePart type="family">Britz</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Melody</namePart>
        <namePart type="family">Guan</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Minh-Thang</namePart>
        <namePart type="family">Luong</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2017-09</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Martha</namePart>
            <namePart type="family">Palmer</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Rebecca</namePart>
            <namePart type="family">Hwa</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Sebastian</namePart>
            <namePart type="family">Riedel</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Copenhagen, Denmark</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>The standard content-based attention mechanism typically used in sequence-to-sequence models is computationally expensive as it requires the comparison of large encoder and decoder states at each time step. In this work, we propose an alternative attention mechanism based on a fixed size memory representation that is more efficient. Our technique predicts a compact set of K attention contexts during encoding and lets the decoder compute an efficient lookup that does not need to consult the memory. We show that our approach performs on-par with the standard attention mechanism while yielding inference speedups of 20% for real-world translation tasks and more for tasks with longer sequences. By visualizing attention scores we demonstrate that our models learn distinct, meaningful alignments.</abstract>
    <identifier type="citekey">britz-etal-2017-efficient</identifier>
    <identifier type="doi">10.18653/v1/D17-1040</identifier>
    <location>
        <url>https://aclanthology.org/D17-1040</url>
    </location>
    <part>
        <date>2017-09</date>
        <extent unit="page">
            <start>392</start>
            <end>400</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Efficient Attention using a Fixed-Size Memory Representation
%A Britz, Denny
%A Guan, Melody
%A Luong, Minh-Thang
%Y Palmer, Martha
%Y Hwa, Rebecca
%Y Riedel, Sebastian
%S Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing
%D 2017
%8 September
%I Association for Computational Linguistics
%C Copenhagen, Denmark
%F britz-etal-2017-efficient
%X The standard content-based attention mechanism typically used in sequence-to-sequence models is computationally expensive as it requires the comparison of large encoder and decoder states at each time step. In this work, we propose an alternative attention mechanism based on a fixed size memory representation that is more efficient. Our technique predicts a compact set of K attention contexts during encoding and lets the decoder compute an efficient lookup that does not need to consult the memory. We show that our approach performs on-par with the standard attention mechanism while yielding inference speedups of 20% for real-world translation tasks and more for tasks with longer sequences. By visualizing attention scores we demonstrate that our models learn distinct, meaningful alignments.
%R 10.18653/v1/D17-1040
%U https://aclanthology.org/D17-1040
%U https://doi.org/10.18653/v1/D17-1040
%P 392-400

Markdown (Informal)

[Efficient Attention using a Fixed-Size Memory Representation](https://aclanthology.org/D17-1040) (Britz et al., EMNLP 2017)

• Efficient Attention using a Fixed-Size Memory Representation (Britz et al., EMNLP 2017)

ACL

• Denny Britz, Melody Guan, and Minh-Thang Luong. 2017. Efficient Attention using a Fixed-Size Memory Representation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 392–400, Copenhagen, Denmark. Association for Computational Linguistics.