Synchronous Syntactic Attention for Transformer Neural Machine Translation

Hiroyuki Deguchi, Akihiro Tamura, Takashi Ninomiya


Abstract
This paper proposes a novel attention mechanism for Transformer Neural Machine Translation, “Synchronous Syntactic Attention,” inspired by synchronous dependency grammars. The mechanism synchronizes source-side and target-side syntactic self-attentions by minimizing the difference between target-side self-attentions and the source-side self-attentions mapped by the encoder-decoder attention matrix. The experiments show that the proposed method improves the translation performance on WMT14 En-De, WMT16 En-Ro, and ASPEC Ja-En (up to +0.38 points in BLEU).
Anthology ID:
2021.acl-srw.36
Volume:
Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop
Month:
August
Year:
2021
Address:
Online
Venues:
ACL | IJCNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
348–355
Language:
URL:
https://aclanthology.org/2021.acl-srw.36
DOI:
10.18653/v1/2021.acl-srw.36
Bibkey:
Cite (ACL):
Hiroyuki Deguchi, Akihiro Tamura, and Takashi Ninomiya. 2021. Synchronous Syntactic Attention for Transformer Neural Machine Translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop, pages 348–355, Online. Association for Computational Linguistics.
Cite (Informal):
Synchronous Syntactic Attention for Transformer Neural Machine Translation (Deguchi et al., ACL 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.acl-srw.36.pdf