Self-Supervised Learning for Pairwise Data Refinement

Gustavo Hernandez Abrego, Bowen Liang, Wei Wang, Zarana Parekh, Yinfei Yang, Yunhsuan Sung


Abstract
Pairwise data automatically constructed from weakly supervised signals has been widely used for training deep learning models. Pairwise datasets such as parallel texts can have uneven quality levels overall, but usually contain data subsets that are more useful as learning examples. We present two methods to refine data that are aimed to obtain that kind of subsets in a self-supervised way. Our methods are based on iteratively training dual-encoder models to compute similarity scores. We evaluate our methods on de-noising parallel texts and training neural machine translation models. We find that: (i) The self-supervised refinement achieves most machine translation gains in the first iteration, but following iterations further improve its intrinsic evaluation. (ii) Machine translations can improve the de-noising performance when combined with selection steps. (iii) Our methods are able to reach the performance of a supervised method. Being entirely self-supervised, our methods are well-suited to handle pairwise data without the need of prior knowledge or human annotations.
Anthology ID:
2020.aacl-main.45
Volume:
Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing
Month:
December
Year:
2020
Address:
Suzhou, China
Venue:
AACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
435–446
Language:
URL:
https://aclanthology.org/2020.aacl-main.45
DOI:
Bibkey:
Cite (ACL):
Gustavo Hernandez Abrego, Bowen Liang, Wei Wang, Zarana Parekh, Yinfei Yang, and Yunhsuan Sung. 2020. Self-Supervised Learning for Pairwise Data Refinement. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 435–446, Suzhou, China. Association for Computational Linguistics.
Cite (Informal):
Self-Supervised Learning for Pairwise Data Refinement (Hernandez Abrego et al., AACL 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.aacl-main.45.pdf