Effective Fine-Tuning Methods for Cross-lingual Adaptation

Tao Yu, Shafiq Joty


Abstract
Large scale multilingual pre-trained language models have shown promising results in zero- and few-shot cross-lingual tasks. However, recent studies have shown their lack of generalizability when the languages are structurally dissimilar. In this work, we propose a novel fine-tuning method based on co-training that aims to learn more generalized semantic equivalences as a complementary to multilingual language modeling using the unlabeled data in the target language. We also propose an adaption method based on contrastive learning to better capture the semantic relationship in the parallel data, when a few translation pairs are available. To show our method’s effectiveness, we conduct extensive experiments on cross-lingual inference and review classification tasks across various languages. We report significant gains compared to directly fine-tuning multilingual pre-trained models and other semi-supervised alternatives.
Anthology ID:
2021.emnlp-main.668
Volume:
Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
Month:
November
Year:
2021
Address:
Online and Punta Cana, Dominican Republic
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
8492–8501
Language:
URL:
https://aclanthology.org/2021.emnlp-main.668
DOI:
10.18653/v1/2021.emnlp-main.668
Bibkey:
Cite (ACL):
Tao Yu and Shafiq Joty. 2021. Effective Fine-Tuning Methods for Cross-lingual Adaptation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8492–8501, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Cite (Informal):
Effective Fine-Tuning Methods for Cross-lingual Adaptation (Yu & Joty, EMNLP 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.emnlp-main.668.pdf
Video:
 https://aclanthology.org/2021.emnlp-main.668.mp4
Data
XNLI