Diverse Lottery Tickets Boost Ensemble from a Single Pretrained Model

Sosuke Kobayashi, Shun Kiyono, Jun Suzuki, Kentaro Inui


Abstract
Ensembling is a popular method used to improve performance as a last resort. However, ensembling multiple models finetuned from a single pretrained model has been not very effective; this could be due to the lack of diversity among ensemble members. This paper proposes Multi-Ticket Ensemble, which finetunes different subnetworks of a single pretrained model and ensembles them. We empirically demonstrated that winning-ticket subnetworks produced more diverse predictions than dense networks and their ensemble outperformed the standard ensemble in some tasks when accurate lottery tickets are found on the tasks.
Anthology ID:
2022.bigscience-1.4
Volume:
Proceedings of BigScience Episode #5 -- Workshop on Challenges & Perspectives in Creating Large Language Models
Month:
May
Year:
2022
Address:
virtual+Dublin
Editors:
Angela Fan, Suzana Ilic, Thomas Wolf, Matthias Gallé
Venue:
BigScience
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
42–50
Language:
URL:
https://aclanthology.org/2022.bigscience-1.4
DOI:
10.18653/v1/2022.bigscience-1.4
Bibkey:
Cite (ACL):
Sosuke Kobayashi, Shun Kiyono, Jun Suzuki, and Kentaro Inui. 2022. Diverse Lottery Tickets Boost Ensemble from a Single Pretrained Model. In Proceedings of BigScience Episode #5 -- Workshop on Challenges & Perspectives in Creating Large Language Models, pages 42–50, virtual+Dublin. Association for Computational Linguistics.
Cite (Informal):
Diverse Lottery Tickets Boost Ensemble from a Single Pretrained Model (Kobayashi et al., BigScience 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.bigscience-1.4.pdf
Video:
 https://aclanthology.org/2022.bigscience-1.4.mp4
Data
GLUEMRPCSQuADSSTSST-2SuperGLUE