Breaking Down Multilingual Machine Translation

Ting-Rui Chiang, Yi-Pei Chen, Yi-Ting Yeh, Graham Neubig


Abstract
While multilingual training is now an essential ingredient in machine translation (MT) systems, recent work has demonstrated that it has different effects in different multilingual settings, such as many-to-one, one-to-many, and many-to-many learning. These training settings expose the encoder and the decoder in a machine translation model with different data distributions. In this paper, we examine how different varieties of multilingual training contribute to learning these two components of the MT model. Specifically, we compare bilingual models with encoders and/or decoders initialized by multilingual training. We show that multilingual training is beneficial to encoders in general, while it only benefits decoders for low-resource languages (LRLs). We further find the important attention heads for each language pair and compare their correlations during inference. Our analysis sheds light on how multilingual translation models work and also enables us to propose methods to improve performance by training with highly related languages. Our many-to-one models for high-resource languages and one-to-many models for LRL outperform the best results reported by Aharoni et al. (2019).
Anthology ID:
2022.findings-acl.218
Volume:
Findings of the Association for Computational Linguistics: ACL 2022
Month:
May
Year:
2022
Address:
Dublin, Ireland
Editors:
Smaranda Muresan, Preslav Nakov, Aline Villavicencio
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2766–2780
Language:
URL:
https://aclanthology.org/2022.findings-acl.218
DOI:
10.18653/v1/2022.findings-acl.218
Bibkey:
Cite (ACL):
Ting-Rui Chiang, Yi-Pei Chen, Yi-Ting Yeh, and Graham Neubig. 2022. Breaking Down Multilingual Machine Translation. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2766–2780, Dublin, Ireland. Association for Computational Linguistics.
Cite (Informal):
Breaking Down Multilingual Machine Translation (Chiang et al., Findings 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.findings-acl.218.pdf