Synchronous Syntactic Attention for Transformer Neural Machine Translation

Hiroyuki Deguchi, Akihiro Tamura, Takashi Ninomiya


Abstract
This paper proposes a novel attention mechanism for Transformer Neural Machine Translation, “Synchronous Syntactic Attention,” inspired by synchronous dependency grammars. The mechanism synchronizes source-side and target-side syntactic self-attentions by minimizing the difference between target-side self-attentions and the source-side self-attentions mapped by the encoder-decoder attention matrix. The experiments show that the proposed method improves the translation performance on WMT14 En-De, WMT16 En-Ro, and ASPEC Ja-En (up to +0.38 points in BLEU).
Anthology ID:
2021.acl-srw.36
Volume:
Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop
Month:
August
Year:
2021
Address:
Online
Venues:
ACL | IJCNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
348–355
Language:
URL:
https://aclanthology.org/2021.acl-srw.36
DOI:
10.18653/v1/2021.acl-srw.36
Bibkey:
Copy Citation:
PDF:
https://aclanthology.org/2021.acl-srw.36.pdf