<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="xue-aletras-2023-pit">
    <titleInfo>
        <title>Pit One Against Many: Leveraging Attention-head Embeddings for Parameter-efficient Multi-head Attention</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Huiyin</namePart>
        <namePart type="family">Xue</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Nikolaos</namePart>
        <namePart type="family">Aletras</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: EMNLP 2023</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Houda</namePart>
            <namePart type="family">Bouamor</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Juan</namePart>
            <namePart type="family">Pino</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Kalika</namePart>
            <namePart type="family">Bali</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Singapore</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Scaling pre-trained language models has resulted in large performance gains in various natural language processing tasks but comes with a large cost in memory requirements. Inspired by the position embeddings in transformers, we aim to simplify and reduce the memory footprint of the multi-head attention (MHA) mechanism. We propose an alternative module that uses only a single shared projection matrix and multiple head embeddings (MHE), i.e. one per head. We empirically demonstrate that our MHE attention is substantially more memory efficient compared to alternative attention mechanisms while achieving high predictive performance retention ratio to vanilla MHA on several downstream tasks. MHE attention only requires a negligible fraction of additional parameters (3nd, where n is the number of attention heads and d the size of the head embeddings) compared to a single-head attention, while MHA requires (3n²-3n)d²-3nd additional parameters.</abstract>
    <identifier type="citekey">xue-aletras-2023-pit</identifier>
    <identifier type="doi">10.18653/v1/2023.findings-emnlp.695</identifier>
    <location>
        <url>https://aclanthology.org/2023.findings-emnlp.695</url>
    </location>
    <part>
        <date>2023-12</date>
        <extent unit="page">
            <start>10355</start>
            <end>10373</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Pit One Against Many: Leveraging Attention-head Embeddings for Parameter-efficient Multi-head Attention
%A Xue, Huiyin
%A Aletras, Nikolaos
%Y Bouamor, Houda
%Y Pino, Juan
%Y Bali, Kalika
%S Findings of the Association for Computational Linguistics: EMNLP 2023
%D 2023
%8 December
%I Association for Computational Linguistics
%C Singapore
%F xue-aletras-2023-pit
%X Scaling pre-trained language models has resulted in large performance gains in various natural language processing tasks but comes with a large cost in memory requirements. Inspired by the position embeddings in transformers, we aim to simplify and reduce the memory footprint of the multi-head attention (MHA) mechanism. We propose an alternative module that uses only a single shared projection matrix and multiple head embeddings (MHE), i.e. one per head. We empirically demonstrate that our MHE attention is substantially more memory efficient compared to alternative attention mechanisms while achieving high predictive performance retention ratio to vanilla MHA on several downstream tasks. MHE attention only requires a negligible fraction of additional parameters (3nd, where n is the number of attention heads and d the size of the head embeddings) compared to a single-head attention, while MHA requires (3n²-3n)d²-3nd additional parameters.
%R 10.18653/v1/2023.findings-emnlp.695
%U https://aclanthology.org/2023.findings-emnlp.695
%U https://doi.org/10.18653/v1/2023.findings-emnlp.695
%P 10355-10373

Markdown (Informal)

[Pit One Against Many: Leveraging Attention-head Embeddings for Parameter-efficient Multi-head Attention](https://aclanthology.org/2023.findings-emnlp.695) (Xue & Aletras, Findings 2023)

• Pit One Against Many: Leveraging Attention-head Embeddings for Parameter-efficient Multi-head Attention (Xue & Aletras, Findings 2023)

ACL

• Huiyin Xue and Nikolaos Aletras. 2023. Pit One Against Many: Leveraging Attention-head Embeddings for Parameter-efficient Multi-head Attention. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10355–10373, Singapore. Association for Computational Linguistics.