Go Simple and Pre-Train on Domain-Specific Corpora: On the Role of Training Data for Text Classification

Aleksandra Edwards, Jose Camacho-Collados, Hélène De Ribaupierre, Alun Preece


Abstract
Pre-trained language models provide the foundations for state-of-the-art performance across a wide range of natural language processing tasks, including text classification. However, most classification datasets assume a large amount labeled data, which is commonly not the case in practical settings. In particular, in this paper we compare the performance of a light-weight linear classifier based on word embeddings, i.e., fastText (Joulin et al., 2017), versus a pre-trained language model, i.e., BERT (Devlin et al., 2019), across a wide range of datasets and classification tasks. In general, results show the importance of domain-specific unlabeled data, both in the form of word embeddings or language models. As for the comparison, BERT outperforms all baselines in standard datasets with large training sets. However, in settings with small training datasets a simple method like fastText coupled with domain-specific word embeddings performs equally well or better than BERT, even when pre-trained on domain-specific data.
Anthology ID:
2020.coling-main.481
Volume:
Proceedings of the 28th International Conference on Computational Linguistics
Month:
December
Year:
2020
Address:
Barcelona, Spain (Online)
Editors:
Donia Scott, Nuria Bel, Chengqing Zong
Venue:
COLING
SIG:
Publisher:
International Committee on Computational Linguistics
Note:
Pages:
5522–5529
Language:
URL:
https://aclanthology.org/2020.coling-main.481
DOI:
10.18653/v1/2020.coling-main.481
Bibkey:
Cite (ACL):
Aleksandra Edwards, Jose Camacho-Collados, Hélène De Ribaupierre, and Alun Preece. 2020. Go Simple and Pre-Train on Domain-Specific Corpora: On the Role of Training Data for Text Classification. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5522–5529, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Cite (Informal):
Go Simple and Pre-Train on Domain-Specific Corpora: On the Role of Training Data for Text Classification (Edwards et al., COLING 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.coling-main.481.pdf
Data
AG NewsIMDb Movie Reviews