Temporal Adaptation of BERT and Performance on Downstream Document Classification: Insights from Social Media

Paul Röttger, Janet Pierrehumbert


Abstract
Language use differs between domains and even within a domain, language use changes over time. For pre-trained language models like BERT, domain adaptation through continued pre-training has been shown to improve performance on in-domain downstream tasks. In this article, we investigate whether temporal adaptation can bring additional benefits. For this purpose, we introduce a corpus of social media comments sampled over three years. It contains unlabelled data for adaptation and evaluation on an upstream masked language modelling task as well as labelled data for fine-tuning and evaluation on a downstream document classification task. We find that temporality matters for both tasks: temporal adaptation improves upstream and temporal fine-tuning downstream task performance. Time-specific models generally perform better on past than on future test sets, which matches evidence on the bursty usage of topical words. However, adapting BERT to time and domain does not improve performance on the downstream task over only adapting to domain. Token-level analysis shows that temporal adaptation captures event-driven changes in language use in the downstream task, but not those changes that are actually relevant to task performance. Based on our findings, we discuss when temporal adaptation may be more effective.
Anthology ID:
2021.findings-emnlp.206
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2021
Month:
November
Year:
2021
Address:
Punta Cana, Dominican Republic
Editors:
Marie-Francine Moens, Xuanjing Huang, Lucia Specia, Scott Wen-tau Yih
Venue:
Findings
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
2400–2412
Language:
URL:
https://aclanthology.org/2021.findings-emnlp.206
DOI:
10.18653/v1/2021.findings-emnlp.206
Bibkey:
Cite (ACL):
Paul Röttger and Janet Pierrehumbert. 2021. Temporal Adaptation of BERT and Performance on Downstream Document Classification: Insights from Social Media. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2400–2412, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Cite (Informal):
Temporal Adaptation of BERT and Performance on Downstream Document Classification: Insights from Social Media (Röttger & Pierrehumbert, Findings 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.findings-emnlp.206.pdf
Video:
 https://aclanthology.org/2021.findings-emnlp.206.mp4
Code
 paul-rottger/temporal-adaptation +  additional community code
Data
RTCPushshift Reddit