Exploiting Curriculum Learning in Unsupervised Neural Machine Translation

Jinliang Lu, Jiajun Zhang


Abstract
Back-translation (BT) has become one of the de facto components in unsupervised neural machine translation (UNMT), and it explicitly makes UNMT have translation ability. However, all the pseudo bi-texts generated by BT are treated equally as clean data during optimization without considering the quality diversity, leading to slow convergence and limited translation performance. To address this problem, we propose a curriculum learning method to gradually utilize pseudo bi-texts based on their quality from multiple granularities. Specifically, we first apply crosslingual word embedding to calculate the potential translation difficulty (quality) for the monolingual sentences. Then, the sentences are fed into UNMT from easy to hard batch by batch. Furthermore, considering the quality of sentences/tokens in a particular batch are also diverse, we further adopt the model itself to calculate the fine-grained quality scores, which are served as learning factors to balance the contributions of different parts when computing loss and encourage the UNMT model to focus on pseudo data with higher quality. Experimental results on WMT 14 En-Fr, WMT 14 En-De, WMT 16 En-Ro, and LDC En-Zh translation tasks demonstrate that the proposed method achieves consistent improvements with faster convergence speed.
Anthology ID:
2021.findings-emnlp.79
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2021
Month:
November
Year:
2021
Address:
Punta Cana, Dominican Republic
Editors:
Marie-Francine Moens, Xuanjing Huang, Lucia Specia, Scott Wen-tau Yih
Venue:
Findings
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
924–934
Language:
URL:
https://aclanthology.org/2021.findings-emnlp.79
DOI:
10.18653/v1/2021.findings-emnlp.79
Bibkey:
Cite (ACL):
Jinliang Lu and Jiajun Zhang. 2021. Exploiting Curriculum Learning in Unsupervised Neural Machine Translation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 924–934, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Cite (Informal):
Exploiting Curriculum Learning in Unsupervised Neural Machine Translation (Lu & Zhang, Findings 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.findings-emnlp.79.pdf
Video:
 https://aclanthology.org/2021.findings-emnlp.79.mp4
Code
 jinlianglu96/cl_unmt