<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="tang-etal-2018-analysis">
    <titleInfo>
        <title>An Analysis of Attention Mechanisms: The Case of Word Sense Disambiguation in Neural Machine Translation</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Gongbo</namePart>
        <namePart type="family">Tang</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Rico</namePart>
        <namePart type="family">Sennrich</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Joakim</namePart>
        <namePart type="family">Nivre</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2018-10</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the Third Conference on Machine Translation: Research Papers</title>
        </titleInfo>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Brussels, Belgium</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Recent work has shown that the encoder-decoder attention mechanisms in neural machine translation (NMT) are different from the word alignment in statistical machine translation. In this paper, we focus on analyzing encoder-decoder attention mechanisms, in the case of word sense disambiguation (WSD) in NMT models. We hypothesize that attention mechanisms pay more attention to context tokens when translating ambiguous words. We explore the attention distribution patterns when translating ambiguous nouns. Counterintuitively, we find that attention mechanisms are likely to distribute more attention to the ambiguous noun itself rather than context tokens, in comparison to other nouns. We conclude that attention is not the main mechanism used by NMT models to incorporate contextual information for WSD. The experimental results suggest that NMT models learn to encode contextual information necessary for WSD in the encoder hidden states. For the attention mechanism in Transformer models, we reveal that the first few layers gradually learn to “align” source and target tokens and the last few layers learn to extract features from the related but unaligned context tokens.</abstract>
    <identifier type="citekey">tang-etal-2018-analysis</identifier>
    <identifier type="doi">10.18653/v1/W18-6304</identifier>
    <location>
        <url>https://aclanthology.org/W18-6304</url>
    </location>
    <part>
        <date>2018-10</date>
        <extent unit="page">
            <start>26</start>
            <end>35</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T An Analysis of Attention Mechanisms: The Case of Word Sense Disambiguation in Neural Machine Translation
%A Tang, Gongbo
%A Sennrich, Rico
%A Nivre, Joakim
%S Proceedings of the Third Conference on Machine Translation: Research Papers
%D 2018
%8 October
%I Association for Computational Linguistics
%C Brussels, Belgium
%F tang-etal-2018-analysis
%X Recent work has shown that the encoder-decoder attention mechanisms in neural machine translation (NMT) are different from the word alignment in statistical machine translation. In this paper, we focus on analyzing encoder-decoder attention mechanisms, in the case of word sense disambiguation (WSD) in NMT models. We hypothesize that attention mechanisms pay more attention to context tokens when translating ambiguous words. We explore the attention distribution patterns when translating ambiguous nouns. Counterintuitively, we find that attention mechanisms are likely to distribute more attention to the ambiguous noun itself rather than context tokens, in comparison to other nouns. We conclude that attention is not the main mechanism used by NMT models to incorporate contextual information for WSD. The experimental results suggest that NMT models learn to encode contextual information necessary for WSD in the encoder hidden states. For the attention mechanism in Transformer models, we reveal that the first few layers gradually learn to “align” source and target tokens and the last few layers learn to extract features from the related but unaligned context tokens.
%R 10.18653/v1/W18-6304
%U https://aclanthology.org/W18-6304
%U https://doi.org/10.18653/v1/W18-6304
%P 26-35

Markdown (Informal)

[An Analysis of Attention Mechanisms: The Case of Word Sense Disambiguation in Neural Machine Translation](https://aclanthology.org/W18-6304) (Tang et al., WMT 2018)

• An Analysis of Attention Mechanisms: The Case of Word Sense Disambiguation in Neural Machine Translation (Tang et al., WMT 2018)

ACL

• Gongbo Tang, Rico Sennrich, and Joakim Nivre. 2018. An Analysis of Attention Mechanisms: The Case of Word Sense Disambiguation in Neural Machine Translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 26–35, Brussels, Belgium. Association for Computational Linguistics.