GOLD: Generalized Knowledge Distillation via Out-of-Distribution-Guided Language Data Generation

Mohsen Gholami, Mohammad Akbari, Tianxi Hu, Vaden Masrani, Z. Wang, Yong Zhang


Abstract
Knowledge distillation from LLMs is essential for the efficient deployment of language models. Prior works have proposed data generation using LLMs for preparing distilled models. We argue that generating data with LLMs is prone to sampling mainly from the center of original content distribution. This limitation hinders the distilled model from learning the true underlying data distribution and to forget the tails of the distributions (samples with lower probability). To this end, we propose GOLD, a task-agnostic data generation and knowledge distillation framework, which employs an iterative out-of-distribution-guided feedback mechanism for the LLM. As a result, the generated data improves the generalizability of distilled models. An energy-based OOD evaluation approach is also introduced to deal with noisy generated data. Our extensive experiments on 10 different classification and sequence-to-sequence tasks in NLP show that GOLD respectively outperforms prior arts and the LLM with an average improvement of 5% and 14%. We will also show that the proposed method is applicable to less explored and novel tasks. Code is available in the Appendix.
Anthology ID:
2024.findings-naacl.272
Volume:
Findings of the Association for Computational Linguistics: NAACL 2024
Month:
June
Year:
2024
Address:
Mexico City, Mexico
Editors:
Kevin Duh, Helena Gomez, Steven Bethard
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
4365–4380
Language:
URL:
https://aclanthology.org/2024.findings-naacl.272
DOI:
Bibkey:
Cite (ACL):
Mohsen Gholami, Mohammad Akbari, Tianxi Hu, Vaden Masrani, Z. Wang, and Yong Zhang. 2024. GOLD: Generalized Knowledge Distillation via Out-of-Distribution-Guided Language Data Generation. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 4365–4380, Mexico City, Mexico. Association for Computational Linguistics.
Cite (Informal):
GOLD: Generalized Knowledge Distillation via Out-of-Distribution-Guided Language Data Generation (Gholami et al., Findings 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.findings-naacl.272.pdf
Copyright:
 2024.findings-naacl.272.copyright.pdf