@inproceedings{shuaibo-etal-2022-supervised,
    title = "Supervised Contrastive Learning for Cross-lingual Transfer Learning",
    author = "Shuaibo, Wang  and
      Hui, Di  and
      Hui, Huang  and
      Siyu, Lai  and
      Kazushige, Ouchi  and
      Yufeng, Chen  and
      Jinan, Xu",
    booktitle = "Proceedings of the 21st Chinese National Conference on Computational Linguistics",
    month = oct,
    year = "2022",
    address = "Nanchang, China",
    publisher = "Chinese Information Processing Society of China",
    url = "https://aclanthology.org/2022.ccl-1.78",
    pages = "884--895",
    abstract = "{``}Multilingual pre-trained representations are not well-aligned by nature, which harms their performance on cross-lingual tasks. Previous methods propose to post-align the multilingual pretrained representations by multi-view alignment or contrastive learning. However, we argue that both methods are not suitable for the cross-lingual classification objective, and in this paper we propose a simple yet effective method to better align the pre-trained representations. On the basis of cross-lingual data augmentations, we make a minor modification to the canonical contrastive loss, to remove false-negative examples which should not be contrasted. Augmentations with the same class are brought close to the anchor sample, and augmentations with different class are pushed apart. Experiment results on three cross-lingual tasks from XTREME benchmark show our method could improve the transfer performance by a large margin with no additional resource needed. We also provide in-detail analysis and comparison between different post-alignment strategies.{''}",
    language = "English",
}