GhostBERT: Generate More Features with Cheap Operations for BERT

Zhiqi Huang, Lu Hou, Lifeng Shang, Xin Jiang, Xiao Chen, Qun Liu


Abstract
Transformer-based pre-trained language models like BERT, though powerful in many tasks, are expensive in both memory and computation, due to their large number of parameters. Previous works show that some parameters in these models can be pruned away without severe accuracy drop. However, these redundant features contribute to a comprehensive understanding of the training data and removing them weakens the model’s representation ability. In this paper, we propose GhostBERT, which generates more features with very cheap operations from the remaining features. In this way, GhostBERT has similar memory and computational cost as the pruned model, but enjoys much larger representation power. The proposed ghost module can also be applied to unpruned BERT models to enhance their performance with negligible additional parameters and computation. Empirical results on the GLUE benchmark on three backbone models (i.e., BERT, RoBERTa and ELECTRA) verify the efficacy of our proposed method.
Anthology ID:
2021.acl-long.509
Volume:
Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)
Month:
August
Year:
2021
Address:
Online
Venues:
ACL | IJCNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
6512–6523
Language:
URL:
https://aclanthology.org/2021.acl-long.509
DOI:
10.18653/v1/2021.acl-long.509
Bibkey:
Cite (ACL):
Zhiqi Huang, Lu Hou, Lifeng Shang, Xin Jiang, Xiao Chen, and Qun Liu. 2021. GhostBERT: Generate More Features with Cheap Operations for BERT. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6512–6523, Online. Association for Computational Linguistics.
Cite (Informal):
GhostBERT: Generate More Features with Cheap Operations for BERT (Huang et al., ACL 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.acl-long.509.pdf
Video:
 https://aclanthology.org/2021.acl-long.509.mp4
Data
GLUEQNLI