Combining Compressions for Multiplicative Size Scaling on Natural Language Tasks

Rajiv Movva, Jinhao Lei, Shayne Longpre, Ajay Gupta, Chris DuBois


Abstract
Quantization, knowledge distillation, and magnitude pruning are among the most popular methods for neural network compression in NLP. Independently, these methods reduce model size and can accelerate inference, but their relative benefit and combinatorial interactions have not been rigorously studied. For each of the eight possible subsets of these techniques, we compare accuracy vs. model size tradeoffs across six BERT architecture sizes and eight GLUE tasks. We find that quantization and distillation consistently provide greater benefit than pruning. Surprisingly, except for the pair of pruning and quantization, using multiple methods together rarely yields diminishing returns. Instead, we observe complementary and super-multiplicative reductions to model size. Our work quantitatively demonstrates that combining compression methods can synergistically reduce model size, and that practitioners should prioritize (1) quantization, (2) knowledge distillation, and (3) pruning to maximize accuracy vs. model size tradeoffs.
Anthology ID:
2022.coling-1.252
Volume:
Proceedings of the 29th International Conference on Computational Linguistics
Month:
October
Year:
2022
Address:
Gyeongju, Republic of Korea
Venue:
COLING
SIG:
Publisher:
International Committee on Computational Linguistics
Note:
Pages:
2861–2872
Language:
URL:
https://aclanthology.org/2022.coling-1.252
DOI:
Bibkey:
Cite (ACL):
Rajiv Movva, Jinhao Lei, Shayne Longpre, Ajay Gupta, and Chris DuBois. 2022. Combining Compressions for Multiplicative Size Scaling on Natural Language Tasks. In Proceedings of the 29th International Conference on Computational Linguistics, pages 2861–2872, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Cite (Informal):
Combining Compressions for Multiplicative Size Scaling on Natural Language Tasks (Movva et al., COLING 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.coling-1.252.pdf
Data
GLUEQNLI