Intermediate Self-supervised Learning for Machine Translation Quality Estimation

Raphael Rubino, Eiichiro Sumita


Abstract
Pre-training sentence encoders is effective in many natural language processing tasks including machine translation (MT) quality estimation (QE), due partly to the scarcity of annotated QE data required for supervised learning. In this paper, we investigate the use of an intermediate self-supervised learning task for sentence encoder aiming at improving QE performances at the sentence and word levels. Our approach is motivated by a problem inherent to QE: mistakes in translation caused by wrongly inserted and deleted tokens. We modify the translation language model (TLM) training objective of the cross-lingual language model (XLM) to orientate the pre-trained model towards the target task. The proposed method does not rely on annotated data and is complementary to QE methods involving pre-trained sentence encoders and domain adaptation. Experiments on English-to-German and English-to-Russian translation directions show that intermediate learning improves over domain adaptated models. Additionally, our method reaches results in par with state-of-the-art QE models without requiring the combination of several approaches and outperforms similar methods based on pre-trained sentence encoders.
Anthology ID:
2020.coling-main.385
Volume:
Proceedings of the 28th International Conference on Computational Linguistics
Month:
December
Year:
2020
Address:
Barcelona, Spain (Online)
Editors:
Donia Scott, Nuria Bel, Chengqing Zong
Venue:
COLING
SIG:
Publisher:
International Committee on Computational Linguistics
Note:
Pages:
4355–4360
Language:
URL:
https://aclanthology.org/2020.coling-main.385
DOI:
10.18653/v1/2020.coling-main.385
Bibkey:
Cite (ACL):
Raphael Rubino and Eiichiro Sumita. 2020. Intermediate Self-supervised Learning for Machine Translation Quality Estimation. In Proceedings of the 28th International Conference on Computational Linguistics, pages 4355–4360, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Cite (Informal):
Intermediate Self-supervised Learning for Machine Translation Quality Estimation (Rubino & Sumita, COLING 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.coling-main.385.pdf