Embedding Recycling for Language Models

Jon Saad-Falcon, Amanpreet Singh, Luca Soldaini, Mike D’Arcy, Arman Cohan, Doug Downey


Abstract
Real-world applications of neural language models often involve running many different models over the same corpus. The high computational cost of these runs has led to interest in techniques that can reuse the contextualized embeddings produced in previous runs to speed training and inference of future ones. We refer to this approach as embedding recycling (ER). While multiple ER techniques have been proposed, their practical effectiveness is still unknown because existing evaluations consider very few models and do not adequately account for overhead costs. We perform an extensive evaluation of ER across eight different models (17 to 900 million parameters) and fourteen tasks in English. We show how a simple ER technique that caches activations from an intermediate layer of a pretrained model, and learns task-specific adapters on the later layers, is broadly effective. For the best-performing baseline in our experiments (DeBERTa-v2 XL), adding a precomputed cache results in a 90% speedup during training and 87-91% speedup for inference, with negligible impact on accuracy. Our analysis reveals important areas of future work.
Anthology ID:
2023.findings-eacl.145
Volume:
Findings of the Association for Computational Linguistics: EACL 2023
Month:
May
Year:
2023
Address:
Dubrovnik, Croatia
Editors:
Andreas Vlachos, Isabelle Augenstein
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1933–1953
Language:
URL:
https://aclanthology.org/2023.findings-eacl.145
DOI:
10.18653/v1/2023.findings-eacl.145
Bibkey:
Cite (ACL):
Jon Saad-Falcon, Amanpreet Singh, Luca Soldaini, Mike D’Arcy, Arman Cohan, and Doug Downey. 2023. Embedding Recycling for Language Models. In Findings of the Association for Computational Linguistics: EACL 2023, pages 1933–1953, Dubrovnik, Croatia. Association for Computational Linguistics.
Cite (Informal):
Embedding Recycling for Language Models (Saad-Falcon et al., Findings 2023)
Copy Citation:
PDF:
https://aclanthology.org/2023.findings-eacl.145.pdf
Video:
 https://aclanthology.org/2023.findings-eacl.145.mp4