<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="pires-etal-2023-one">
    <titleInfo>
        <title>One Wide Feedforward Is All You Need</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Telmo</namePart>
        <namePart type="family">Pires</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">António</namePart>
        <namePart type="family">Vilarinho Lopes</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Yannick</namePart>
        <namePart type="family">Assogba</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Hendra</namePart>
        <namePart type="family">Setiawan</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the Eighth Conference on Machine Translation</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Philipp</namePart>
            <namePart type="family">Koehn</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Barry</namePart>
            <namePart type="family">Haddow</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Tom</namePart>
            <namePart type="family">Kocmi</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Christof</namePart>
            <namePart type="family">Monz</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Singapore</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>The Transformer architecture has two main non-embedding components: Attention and the Feed Forward Network (FFN). Attention captures interdependencies between words regardless of their position, while the FFN non-linearly transforms each input token independently. In this work we explore the role of the FFN, and find that despite taking up a significant fraction of the model’s parameters, it is highly redundant. Concretely, we are able to substantially reduce the number of parameters with only a modest drop in accuracy by removing the FFN on the decoder layers and sharing a single FFN across the encoder. Finally we scale this architecture back to its original size by increasing the hidden dimension of the shared FFN, achieving substantial gains in both accuracy and latency with respect to the original Transformer Big.</abstract>
    <identifier type="citekey">pires-etal-2023-one</identifier>
    <identifier type="doi">10.18653/v1/2023.wmt-1.98</identifier>
    <location>
        <url>https://aclanthology.org/2023.wmt-1.98</url>
    </location>
    <part>
        <date>2023-12</date>
        <extent unit="page">
            <start>1031</start>
            <end>1044</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T One Wide Feedforward Is All You Need
%A Pires, Telmo
%A Vilarinho Lopes, António
%A Assogba, Yannick
%A Setiawan, Hendra
%Y Koehn, Philipp
%Y Haddow, Barry
%Y Kocmi, Tom
%Y Monz, Christof
%S Proceedings of the Eighth Conference on Machine Translation
%D 2023
%8 December
%I Association for Computational Linguistics
%C Singapore
%F pires-etal-2023-one
%X The Transformer architecture has two main non-embedding components: Attention and the Feed Forward Network (FFN). Attention captures interdependencies between words regardless of their position, while the FFN non-linearly transforms each input token independently. In this work we explore the role of the FFN, and find that despite taking up a significant fraction of the model’s parameters, it is highly redundant. Concretely, we are able to substantially reduce the number of parameters with only a modest drop in accuracy by removing the FFN on the decoder layers and sharing a single FFN across the encoder. Finally we scale this architecture back to its original size by increasing the hidden dimension of the shared FFN, achieving substantial gains in both accuracy and latency with respect to the original Transformer Big.
%R 10.18653/v1/2023.wmt-1.98
%U https://aclanthology.org/2023.wmt-1.98
%U https://doi.org/10.18653/v1/2023.wmt-1.98
%P 1031-1044

Markdown (Informal)

[One Wide Feedforward Is All You Need](https://aclanthology.org/2023.wmt-1.98) (Pires et al., WMT 2023)

• One Wide Feedforward Is All You Need (Pires et al., WMT 2023)

ACL

• Telmo Pires, António Vilarinho Lopes, Yannick Assogba, and Hendra Setiawan. 2023. One Wide Feedforward Is All You Need. In Proceedings of the Eighth Conference on Machine Translation, pages 1031–1044, Singapore. Association for Computational Linguistics.