SynthesizRR: Generating Diverse Datasets with Retrieval Augmentation

Abhishek Divekar, Greg Durrett


Abstract
It is often desirable to distill the capabilities of large language models (LLMs) into smaller student models due to compute and memory constraints. One way to do this for classification tasks is via dataset synthesis, which can be accomplished by generating examples of each label from the LLM. Prior approaches to synthesis use few-shot prompting, which relies on the LLM’s parametric knowledge to generate usable examples. However, this leads to issues of repetition, bias towards popular entities, and stylistic differences from human text. In this work, we propose Synthesize by Retrieval and Refinement (SynthesizRR), which uses retrieval augmentation to introduce variety into the dataset synthesis process: as retrieved passages vary, the LLM is seeded with different content to generate its examples. We empirically study the synthesis of six datasets, covering topic classification, sentiment analysis, tone detection, and humor, requiring complex synthesis strategies. We find SynthesizRR greatly improves lexical and semantic diversity, similarity to human-written text, and distillation performance, when compared to 32-shot prompting and four prior approaches.
Anthology ID:
2024.emnlp-main.1071
Volume:
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
Month:
November
Year:
2024
Address:
Miami, Florida, USA
Editors:
Yaser Al-Onaizan, Mohit Bansal, Yun-Nung Chen
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
19200–19227
Language:
URL:
https://aclanthology.org/2024.emnlp-main.1071
DOI:
10.18653/v1/2024.emnlp-main.1071
Bibkey:
Cite (ACL):
Abhishek Divekar and Greg Durrett. 2024. SynthesizRR: Generating Diverse Datasets with Retrieval Augmentation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 19200–19227, Miami, Florida, USA. Association for Computational Linguistics.
Cite (Informal):
SynthesizRR: Generating Diverse Datasets with Retrieval Augmentation (Divekar & Durrett, EMNLP 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.emnlp-main.1071.pdf
Software:
 2024.emnlp-main.1071.software.zip