An End-to-End Submodular Framework for Data-Efficient In-Context Learning

Lilly Kumari, Shengjie Wang, Arnav Das, Tianyi Zhou, Jeff Bilmes


Abstract
Recent advancements in natural language tasks leverage the emergent In-Context Learning (ICL) ability of pretrained Large Language Models (LLMs). ICL enables LLMs to perform new tasks by utilizing a limited number of input-output examples as prompts. While ICL circumvents the costly step of finetuning LLMs, its effectiveness is heavily dependent on the quality and ordering of provided examples (called exemplars). In this work, we propose a two-stage data-efficient framework Div-S3 for exemplar selection for ICL. The first stage focuses on data annotation and employs a pool-based active learning approach to select a set of Diverse and informative exemplars from the target tasks’ unlabeled pool. Given a test input/query, the second stage uses Submodular Span Summarization (S3) to select the most relevant and non-redundant exemplars from the annotated pool of a limited budget. On 7 different NLP datasets and 5 LLMs of varying complexities, we show Div-S3 outperforms (1) existing active learning-based methods for data annotation for ICL and (2) similarity-based methods for test query-specific exemplars retrieval.
Anthology ID:
2024.findings-naacl.209
Volume:
Findings of the Association for Computational Linguistics: NAACL 2024
Month:
June
Year:
2024
Address:
Mexico City, Mexico
Editors:
Kevin Duh, Helena Gomez, Steven Bethard
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
3293–3308
Language:
URL:
https://aclanthology.org/2024.findings-naacl.209
DOI:
Bibkey:
Cite (ACL):
Lilly Kumari, Shengjie Wang, Arnav Das, Tianyi Zhou, and Jeff Bilmes. 2024. An End-to-End Submodular Framework for Data-Efficient In-Context Learning. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 3293–3308, Mexico City, Mexico. Association for Computational Linguistics.
Cite (Informal):
An End-to-End Submodular Framework for Data-Efficient In-Context Learning (Kumari et al., Findings 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.findings-naacl.209.pdf
Copyright:
 2024.findings-naacl.209.copyright.pdf