”Diversity and Uncertainty in Moderation” are the Key to Data Selection for Multilingual Few-shot Transfer

Shanu Kumar, Sandipan Dandapat, Monojit Choudhury


Abstract
Few-shot transfer often shows substantial gain over zero-shot transfer (CITATION), which is a practically useful trade-off between fully supervised and unsupervised learning approaches for multilingual pretained model-based systems. This paper explores various strategies for selecting data for annotation that can result in a better few-shot transfer. The proposed approaches rely on multiple measures such as data entropy using n-gram language model, predictive entropy, and gradient embedding. We propose a loss embedding method for sequence labeling tasks, which induces diversity and uncertainty sampling similar to gradient embedding. The proposed data selection strategies are evaluated and compared for POS tagging, NER, and NLI tasks for up to 20 languages. Our experiments show that the gradient and loss embedding-based strategies consistently outperform random data selection baselines, with gains varying with the initial performance of the zero-shot transfer. Furthermore, the proposed method shows similar trends in improvement even when the model is fine-tuned using a lower proportion of the original task-specific labeled training data for zero-shot transfer.
Anthology ID:
2022.findings-naacl.78
Volume:
Findings of the Association for Computational Linguistics: NAACL 2022
Month:
July
Year:
2022
Address:
Seattle, United States
Editors:
Marine Carpuat, Marie-Catherine de Marneffe, Ivan Vladimir Meza Ruiz
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1042–1055
Language:
URL:
https://aclanthology.org/2022.findings-naacl.78
DOI:
10.18653/v1/2022.findings-naacl.78
Bibkey:
Cite (ACL):
Shanu Kumar, Sandipan Dandapat, and Monojit Choudhury. 2022. ”Diversity and Uncertainty in Moderation” are the Key to Data Selection for Multilingual Few-shot Transfer. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 1042–1055, Seattle, United States. Association for Computational Linguistics.
Cite (Informal):
”Diversity and Uncertainty in Moderation” are the Key to Data Selection for Multilingual Few-shot Transfer (Kumar et al., Findings 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.findings-naacl.78.pdf
Video:
 https://aclanthology.org/2022.findings-naacl.78.mp4