Rethinking the Exploitation of Monolingual Data for Low-Resource Neural Machine Translation

Jianhui Pang, Baosong Yang*, Derek Fai Wong*, Yu Wan, Dayiheng Liu, Lidia Sam Chao, Jun Xie


Abstract
The utilization of monolingual data has been shown to be a promising strategy for addressing low-resource machine translation problems. Previous studies have demonstrated the effectiveness of techniques such as back-translation and self-supervised objectives, including masked language modeling, causal language modeling, and denoise autoencoding, in improving the performance of machine translation models. However, the manner in which these methods contribute to the success of machine translation tasks and how they can be effectively combined remains an under-researched area. In this study, we carry out a systematic investigation of the effects of these techniques on linguistic properties through the use of probing tasks, including source language comprehension, bilingual word alignment, and translation fluency. We further evaluate the impact of pre-training, back-translation, and multi-task learning on bitexts of varying sizes. Our findings inform the design of more effective pipelines for leveraging monolingual data in extremely low-resource and low-resource machine translation tasks. Experiment results show consistent performance gains in seven translation directions, which provide further support for our conclusions and understanding of the role of monolingual data in machine translation.
Anthology ID:
2024.cl-1.2
Volume:
Computational Linguistics, Volume 50, Issue 1 - March 2024
Month:
March
Year:
2024
Address:
Cambridge, MA
Venue:
CL
SIG:
Publisher:
MIT Press
Note:
Pages:
25–47
Language:
URL:
https://aclanthology.org/2024.cl-1.2
DOI:
10.1162/coli_a_00496
Bibkey:
Cite (ACL):
Jianhui Pang, Baosong Yang*, Derek Fai Wong*, Yu Wan, Dayiheng Liu, Lidia Sam Chao, and Jun Xie. 2024. Rethinking the Exploitation of Monolingual Data for Low-Resource Neural Machine Translation. Computational Linguistics, 50(1):25–47.
Cite (Informal):
Rethinking the Exploitation of Monolingual Data for Low-Resource Neural Machine Translation (Pang et al., CL 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.cl-1.2.pdf