<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="sun-etal-2022-alleviating">
    <titleInfo>
        <title>Alleviating the Inequality of Attention Heads for Neural Machine Translation</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Zewei</namePart>
        <namePart type="family">Sun</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Shujian</namePart>
        <namePart type="family">Huang</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Xinyu</namePart>
        <namePart type="family">Dai</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Jiajun</namePart>
        <namePart type="family">Chen</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2022-10</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 29th International Conference on Computational Linguistics</title>
        </titleInfo>
        <originInfo>
            <publisher>International Committee on Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Gyeongju, Republic of Korea</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Recent studies show that the attention heads in Transformer are not equal. We relate this phenomenon to the imbalance training of multi-head attention and the model dependence on specific heads. To tackle this problem, we propose a simple masking method: HeadMask, in two specific ways. Experiments show that translation improvements are achieved on multiple language pairs. Subsequent empirical analyses also support our assumption and confirm the effectiveness of the method.</abstract>
    <identifier type="citekey">sun-etal-2022-alleviating</identifier>
    <location>
        <url>https://aclanthology.org/2022.coling-1.466</url>
    </location>
    <part>
        <date>2022-10</date>
        <extent unit="page">
            <start>5246</start>
            <end>5250</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Alleviating the Inequality of Attention Heads for Neural Machine Translation
%A Sun, Zewei
%A Huang, Shujian
%A Dai, Xinyu
%A Chen, Jiajun
%S Proceedings of the 29th International Conference on Computational Linguistics
%D 2022
%8 October
%I International Committee on Computational Linguistics
%C Gyeongju, Republic of Korea
%F sun-etal-2022-alleviating
%X Recent studies show that the attention heads in Transformer are not equal. We relate this phenomenon to the imbalance training of multi-head attention and the model dependence on specific heads. To tackle this problem, we propose a simple masking method: HeadMask, in two specific ways. Experiments show that translation improvements are achieved on multiple language pairs. Subsequent empirical analyses also support our assumption and confirm the effectiveness of the method.
%U https://aclanthology.org/2022.coling-1.466
%P 5246-5250

Markdown (Informal)

[Alleviating the Inequality of Attention Heads for Neural Machine Translation](https://aclanthology.org/2022.coling-1.466) (Sun et al., COLING 2022)

• Alleviating the Inequality of Attention Heads for Neural Machine Translation (Sun et al., COLING 2022)

ACL

• Zewei Sun, Shujian Huang, Xinyu Dai, and Jiajun Chen. 2022. Alleviating the Inequality of Attention Heads for Neural Machine Translation. In Proceedings of the 29th International Conference on Computational Linguistics, pages 5246–5250, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.