%0 Conference Proceedings %T Multi-Stage Pre-training for Low-Resource Domain Adaptation %A Zhang, Rong %A Gangi Reddy, Revanth %A Sultan, Md Arafat %A Castelli, Vittorio %A Ferritto, Anthony %A Florian, Radu %A Sarioglu Kayi, Efsun %A Roukos, Salim %A Sil, Avi %A Ward, Todd %Y Webber, Bonnie %Y Cohn, Trevor %Y He, Yulan %Y Liu, Yang %S Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) %D 2020 %8 November %I Association for Computational Linguistics %C Online %F zhang-etal-2020-multi-stage %X Transfer learning techniques are particularly useful for NLP tasks where a sizable amount of high-quality annotated data is difficult to obtain. Current approaches directly adapt a pretrained language model (LM) on in-domain text before fine-tuning to downstream tasks. We show that extending the vocabulary of the LM with domain-specific terms leads to further gains. To a bigger effect, we utilize structure in the unlabeled data to create auxiliary synthetic tasks, which helps the LM transfer to downstream tasks. We apply these approaches incrementally on a pretrained Roberta-large LM and show considerable performance gain on three tasks in the IT domain: Extractive Reading Comprehension, Document Ranking and Duplicate Question Detection. %R 10.18653/v1/2020.emnlp-main.440 %U https://aclanthology.org/2020.emnlp-main.440 %U https://doi.org/10.18653/v1/2020.emnlp-main.440 %P 5461-5468