oBERTa: Improving Sparse Transfer Learning via improved initialization, distillation, and pruning regimes

oBERTa: Improving Sparse Transfer Learning via improved initialization, distillation, and pruning regimes Daniel Campos author Alexandre Marques author Mark Kurtz author Cheng Xiang Zhai author 2023-07 text Proceedings of the Fourth Workshop on Simple and Efficient Natural Language Processing (SustaiNLP) Nafise Sadat Moosavi editor Iryna Gurevych editor Yufang Hou editor Gyuwan Kim editor Young Jin Kim editor Tal Schuster editor Ameeta Agrawal editor Association for Computational Linguistics Toronto, Canada (Hybrid) conference publication campos-etal-2023-oberta 10.18653/v1/2023.sustainlp-1.3 https://aclanthology.org/2023.sustainlp-1.3/ 2023-07 39 58