Task-specific Compression for Multi-task Language Models using Attribution-based Pruning

Nakyeong Yang, Yunah Jang, Hwanhee Lee, Seohyeong Jeong, Kyomin Jung


Abstract
Multi-task language models show outstanding performance for various natural language understanding tasks with only a single model. However, these language models inevitably utilize an unnecessarily large number of model parameters, even when used only for a specific task. In this paper, we propose a novel training-free compression method for multi-task language models using pruning method. Specifically, we use an attribution method to determine which neurons are essential for performing a specific task. We task-specifically prune unimportant neurons and leave only task-specific parameters. Furthermore, we extend our method to be applicable in both low-resource and unsupervised settings. Since our compression method is training-free, it uses little computing resources and does not update the pre-trained parameters of language models, reducing storage space usage. Experimental results on the six widely-used datasets show that our proposed pruning method significantly outperforms baseline pruning methods. In addition, we demonstrate that our method preserves performance even in an unseen domain setting.
Anthology ID:
2023.findings-eacl.43
Volume:
Findings of the Association for Computational Linguistics: EACL 2023
Month:
May
Year:
2023
Address:
Dubrovnik, Croatia
Editors:
Andreas Vlachos, Isabelle Augenstein
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
594–604
Language:
URL:
https://aclanthology.org/2023.findings-eacl.43
DOI:
10.18653/v1/2023.findings-eacl.43
Bibkey:
Cite (ACL):
Nakyeong Yang, Yunah Jang, Hwanhee Lee, Seohyeong Jeong, and Kyomin Jung. 2023. Task-specific Compression for Multi-task Language Models using Attribution-based Pruning. In Findings of the Association for Computational Linguistics: EACL 2023, pages 594–604, Dubrovnik, Croatia. Association for Computational Linguistics.
Cite (Informal):
Task-specific Compression for Multi-task Language Models using Attribution-based Pruning (Yang et al., Findings 2023)
Copy Citation:
PDF:
https://aclanthology.org/2023.findings-eacl.43.pdf
Video:
 https://aclanthology.org/2023.findings-eacl.43.mp4