TextPruner: A Model Pruning Toolkit for Pre-Trained Language Models

Ziqing Yang, Yiming Cui, Zhigang Chen


Abstract
Pre-trained language models have been prevailed in natural language processing and become the backbones of many NLP tasks, but the demands for computational resources have limited their applications. In this paper, we introduce TextPruner, an open-source model pruning toolkit designed for pre-trained language models, targeting fast and easy model compression. TextPruner offers structured post-training pruning methods, including vocabulary pruning and transformer pruning, and can be applied to various models and tasks. We also propose a self-supervised pruning method that can be applied without the labeled data. Our experiments with several NLP tasks demonstrate the ability of TextPruner to reduce the model size without re-training the model.
Anthology ID:
2022.acl-demo.4
Volume:
Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations
Month:
May
Year:
2022
Address:
Dublin, Ireland
Editors:
Valerio Basile, Zornitsa Kozareva, Sanja Stajner
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
35–43
Language:
URL:
https://aclanthology.org/2022.acl-demo.4
DOI:
10.18653/v1/2022.acl-demo.4
Bibkey:
Cite (ACL):
Ziqing Yang, Yiming Cui, and Zhigang Chen. 2022. TextPruner: A Model Pruning Toolkit for Pre-Trained Language Models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 35–43, Dublin, Ireland. Association for Computational Linguistics.
Cite (Informal):
TextPruner: A Model Pruning Toolkit for Pre-Trained Language Models (Yang et al., ACL 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.acl-demo.4.pdf