## Aligned Weight Regularizers for Pruning Pretrained Neural Networks

##### Abstract
Pruning aims to reduce the number of parameters while maintaining performance close to the original network. This work proposes a novel self-distillation based pruning strategy, whereby the representational similarity between the pruned and unpruned versions of the same network is maximized. Unlike previous approaches that treat distillation and pruning separately, we use distillation to inform the pruning criteria, without requiring a separate student network as in knowledge distillation. We show that the proposed cross-correlation objective for self-distilled pruning implicitly encourages sparse solutions, naturally complementing magnitude-based pruning criteria. Experiments on the GLUE and XGLUE benchmarks show that self-distilled pruning increases mono- and cross-lingual language model performance. Self-distilled pruned models also outperform smaller Transformers with an equal number of parameters and are competitive against (6 times) larger distilled networks. We also observe that self-distillation (1) maximizes class separability, (2) increases the signal-to-noise ratio, and (3) converges faster after pruning steps, providing further insights into why self-distilled pruning improves generalization.
Anthology ID:
2022.findings-acl.267
Volume:
Findings of the Association for Computational Linguistics: ACL 2022
Month:
May
Year:
2022
Dublin, Ireland
Venues:
ACL | Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
3391–3401
Language:
URL:
https://aclanthology.org/2022.findings-acl.267
DOI:
10.18653/v1/2022.findings-acl.267
Cite (ACL):
James O’ Neill, Sourav Dutta, and Haytham Assem. 2022. Aligned Weight Regularizers for Pruning Pretrained Neural Networks. In Findings of the Association for Computational Linguistics: ACL 2022, pages 3391–3401, Dublin, Ireland. Association for Computational Linguistics.
Cite (Informal):
Aligned Weight Regularizers for Pruning Pretrained Neural Networks (O’ Neill et al., Findings 2022)
PDF:
https://aclanthology.org/2022.findings-acl.267.pdf