<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="xu-etal-2021-learning-hard-retrieval">
    <titleInfo>
        <title>Learning Hard Retrieval Decoder Attention for Transformers</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Hongfei</namePart>
        <namePart type="family">Xu</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Qiuhui</namePart>
        <namePart type="family">Liu</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Josef</namePart>
        <namePart type="family">van Genabith</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Deyi</namePart>
        <namePart type="family">Xiong</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2021-11</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: EMNLP 2021</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Marie-Francine</namePart>
            <namePart type="family">Moens</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Xuanjing</namePart>
            <namePart type="family">Huang</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Lucia</namePart>
            <namePart type="family">Specia</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Scott</namePart>
            <namePart type="given">Wen-tau</namePart>
            <namePart type="family">Yih</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Punta Cana, Dominican Republic</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>The Transformer translation model is based on the multi-head attention mechanism, which can be parallelized easily. The multi-head attention network performs the scaled dot-product attention function in parallel, empowering the model by jointly attending to information from different representation subspaces at different positions. In this paper, we present an approach to learning a hard retrieval attention where an attention head only attends to one token in the sentence rather than all tokens. The matrix multiplication between attention probabilities and the value sequence in the standard scaled dot-product attention can thus be replaced by a simple and efficient retrieval operation. We show that our hard retrieval attention mechanism is 1.43 times faster in decoding, while preserving translation quality on a wide range of machine translation tasks when used in the decoder self- and cross-attention networks.</abstract>
    <identifier type="citekey">xu-etal-2021-learning-hard-retrieval</identifier>
    <identifier type="doi">10.18653/v1/2021.findings-emnlp.67</identifier>
    <location>
        <url>https://aclanthology.org/2021.findings-emnlp.67</url>
    </location>
    <part>
        <date>2021-11</date>
        <extent unit="page">
            <start>779</start>
            <end>785</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Learning Hard Retrieval Decoder Attention for Transformers
%A Xu, Hongfei
%A Liu, Qiuhui
%A van Genabith, Josef
%A Xiong, Deyi
%Y Moens, Marie-Francine
%Y Huang, Xuanjing
%Y Specia, Lucia
%Y Yih, Scott Wen-tau
%S Findings of the Association for Computational Linguistics: EMNLP 2021
%D 2021
%8 November
%I Association for Computational Linguistics
%C Punta Cana, Dominican Republic
%F xu-etal-2021-learning-hard-retrieval
%X The Transformer translation model is based on the multi-head attention mechanism, which can be parallelized easily. The multi-head attention network performs the scaled dot-product attention function in parallel, empowering the model by jointly attending to information from different representation subspaces at different positions. In this paper, we present an approach to learning a hard retrieval attention where an attention head only attends to one token in the sentence rather than all tokens. The matrix multiplication between attention probabilities and the value sequence in the standard scaled dot-product attention can thus be replaced by a simple and efficient retrieval operation. We show that our hard retrieval attention mechanism is 1.43 times faster in decoding, while preserving translation quality on a wide range of machine translation tasks when used in the decoder self- and cross-attention networks.
%R 10.18653/v1/2021.findings-emnlp.67
%U https://aclanthology.org/2021.findings-emnlp.67
%U https://doi.org/10.18653/v1/2021.findings-emnlp.67
%P 779-785

Markdown (Informal)

[Learning Hard Retrieval Decoder Attention for Transformers](https://aclanthology.org/2021.findings-emnlp.67) (Xu et al., Findings 2021)

• Learning Hard Retrieval Decoder Attention for Transformers (Xu et al., Findings 2021)

ACL

• Hongfei Xu, Qiuhui Liu, Josef van Genabith, and Deyi Xiong. 2021. Learning Hard Retrieval Decoder Attention for Transformers. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 779–785, Punta Cana, Dominican Republic. Association for Computational Linguistics.