The Diminishing Returns of Masked Language Models to Science

Zhi Hong, Aswathy Ajith, James Pauloski, Eamon Duede, Kyle Chard, Ian Foster


Abstract
Transformer-based masked language models such as BERT, trained on general corpora, have shown impressive performance on downstream tasks. It has also been demonstrated that the downstream task performance of such models can be improved by pretraining larger models for longer on more data. In this work, we empirically evaluate the extent to which these results extend to tasks in science. We use 14 domain-specific transformer-based models (including ScholarBERT, a new 770Mparameter science-focused masked language model pretrained on up to 225B tokens) to evaluate the impact of training data, model size, pretraining and finetuning time on 12 downstream scientific tasks. Interestingly, we find that increasing model size, training data, or compute time does not always lead to significant improvements (i.e., >1% F1), if any, in scientific information extraction tasks. We offer possible explanations for this surprising result.
Anthology ID:
2023.findings-acl.82
Volume:
Findings of the Association for Computational Linguistics: ACL 2023
Month:
July
Year:
2023
Address:
Toronto, Canada
Editors:
Anna Rogers, Jordan Boyd-Graber, Naoaki Okazaki
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1270–1283
Language:
URL:
https://aclanthology.org/2023.findings-acl.82
DOI:
10.18653/v1/2023.findings-acl.82
Bibkey:
Cite (ACL):
Zhi Hong, Aswathy Ajith, James Pauloski, Eamon Duede, Kyle Chard, and Ian Foster. 2023. The Diminishing Returns of Masked Language Models to Science. In Findings of the Association for Computational Linguistics: ACL 2023, pages 1270–1283, Toronto, Canada. Association for Computational Linguistics.
Cite (Informal):
The Diminishing Returns of Masked Language Models to Science (Hong et al., Findings 2023)
Copy Citation:
PDF:
https://aclanthology.org/2023.findings-acl.82.pdf