Modularized Multilingual NMT with Fine-grained Interlingua

Sungjun Lim, Yoonjung Choi, Sangha Kim


Abstract
Recently, one popular alternative in Multilingual NMT (MNMT) is modularized MNMT that has both language-specific encoders and decoders. However, due to the absence of layer-sharing, the modularized MNMT failed to produce satisfactory language-independent (Interlingua) features, leading to performance degradation in zero-shot translation. To address this issue, a solution was proposed to share the top of language-specific encoder layers, enabling the successful generation of interlingua features. Nonetheless, it should be noted that this sharing structure does not guarantee the explicit propagation of language-specific features to their respective language-specific decoders. Consequently, to overcome this challenge, we present our modularized MNMT approach, where a modularized encoder is divided into three distinct encoder modules based on different sharing criteria: (1) source language-specific (Encs); (2) universal (Encall); (3) target language-specific (Enct). By employing these sharing strategies, Encall propagates the interlingua features, after which Enct propagates the target language-specific features to the language-specific decoders. Additionally, we suggest the Denoising Bi-path Autoencoder (DBAE) to fortify the Denoising Autoencoder (DAE) by leveraging Enct. For experimental purposes, our training corpus comprises both En-to-Any and Any-to-En directions. We adjust the size of our corpus to simulate both balanced and unbalanced settings. Our method demonstrates an improved average BLEU score by "+2.90” in En-to-Any directions and by "+3.06” in zero-shot compared to other MNMT baselines.
Anthology ID:
2024.naacl-long.328
Volume:
Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)
Month:
June
Year:
2024
Address:
Mexico City, Mexico
Editors:
Kevin Duh, Helena Gomez, Steven Bethard
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
5884–5899
Language:
URL:
https://aclanthology.org/2024.naacl-long.328
DOI:
Bibkey:
Cite (ACL):
Sungjun Lim, Yoonjung Choi, and Sangha Kim. 2024. Modularized Multilingual NMT with Fine-grained Interlingua. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 5884–5899, Mexico City, Mexico. Association for Computational Linguistics.
Cite (Informal):
Modularized Multilingual NMT with Fine-grained Interlingua (Lim et al., NAACL 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.naacl-long.328.pdf
Copyright:
 2024.naacl-long.328.copyright.pdf