An Empirical Investigation of Word Alignment Supervision for Zero-Shot Multilingual Neural Machine Translation

Alessandro Raganato, Raúl Vázquez, Mathias Creutz, Jörg Tiedemann


Abstract
Zero-shot translations is a fascinating feature of Multilingual Neural Machine Translation (MNMT) systems. These MNMT models are usually trained on English-centric data, i.e. English either as the source or target language, and with a language label prepended to the input indicating the target language. However, recent work has highlighted several flaws of these models in zero-shot scenarios where language labels are ignored and the wrong language is generated or different runs show highly unstable results. In this paper, we investigate the benefits of an explicit alignment to language labels in Transformer-based MNMT models in the zero-shot context, by jointly training one cross attention head with word alignment supervision to stress the focus on the target language label. We compare and evaluate several MNMT systems on three multilingual MT benchmarks of different sizes, showing that simply supervising one cross attention head to focus both on word alignments and language labels reduces the bias towards translating into the wrong language, improving the zero-shot performance overall. Moreover, as an additional advantage, we find that our alignment supervision leads to more stable results across different training runs.
Anthology ID:
2021.emnlp-main.664
Volume:
Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
Month:
November
Year:
2021
Address:
Online and Punta Cana, Dominican Republic
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
8449–8456
Language:
URL:
https://aclanthology.org/2021.emnlp-main.664
DOI:
10.18653/v1/2021.emnlp-main.664
Bibkey:
Cite (ACL):
Alessandro Raganato, Raúl Vázquez, Mathias Creutz, and Jörg Tiedemann. 2021. An Empirical Investigation of Word Alignment Supervision for Zero-Shot Multilingual Neural Machine Translation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8449–8456, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Cite (Informal):
An Empirical Investigation of Word Alignment Supervision for Zero-Shot Multilingual Neural Machine Translation (Raganato et al., EMNLP 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.emnlp-main.664.pdf
Video:
 https://aclanthology.org/2021.emnlp-main.664.mp4
Data
OPUS-100WMT 2018