Predictive Model Selection for Transfer Learning in Sequence Labeling Tasks

Parul Awasthy, Bishwaranjan Bhattacharjee, John Kender, Radu Florian


Abstract
Transfer learning is a popular technique to learn a task using less training data and fewer compute resources. However, selecting the correct source model for transfer learning is a challenging task. We demonstrate a novel predictive method that determines which existing source model would minimize error for transfer learning to a given target. This technique does not require learning for prediction, and avoids computational costs of trail-and-error. We have evaluated this technique on nine datasets across diverse domains, including newswire, user forums, air flight booking, cybersecurity news, etc. We show that it per-forms better than existing techniques such as fine-tuning over vanilla BERT, or curriculum learning over the largest dataset on top of BERT, resulting in average F1 score gains in excess of 3%. Moreover, our technique consistently selects the best model using fewer tries.
Anthology ID:
2020.sustainlp-1.15
Volume:
Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing
Month:
November
Year:
2020
Address:
Online
Venue:
sustainlp
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
113–118
Language:
URL:
https://aclanthology.org/2020.sustainlp-1.15
DOI:
10.18653/v1/2020.sustainlp-1.15
Bibkey:
Cite (ACL):
Parul Awasthy, Bishwaranjan Bhattacharjee, John Kender, and Radu Florian. 2020. Predictive Model Selection for Transfer Learning in Sequence Labeling Tasks. In Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing, pages 113–118, Online. Association for Computational Linguistics.
Cite (Informal):
Predictive Model Selection for Transfer Learning in Sequence Labeling Tasks (Awasthy et al., sustainlp 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.sustainlp-1.15.pdf
Video:
 https://slideslive.com/38939437