Multi-Scale Prompt Memory-Augmented Model for Black-Box Scenarios

Xiaojun Kuang, C. L. Philip Chen, Shuzhen Li, Tong Zhang


Abstract
Black-box few-shot text classification handles text classification in limited data without accessing the parameters and gradients of language models (LMs). Existing black-box optimization methods have demonstrated strong few-shot learning capabilities. However, they still require numerous LMs’ calls to search optimal prompts, thus resulting in overfitting performance and increasing computational cost. To address this issue, we present MuSKPrompt (Multi-scale Knowledge Prompt for Memory Model), an efficient multi-scale knowledge prompt-based memory model in black-box few-shot text classification task. MuSKPrompt extracts instance-level and class-level knowledge at different scales and stores them in memory banks during training. Then, it references multi-scale memory banks to perform quick inference on new samples via a novel scoring module. MuSKPrompt achieves competitive performance in limited data through multi-scale instance-level and class-level knowledge. Moreover, it realizes gradient-free optimization with zero training parameters in the black-box scenario. Experiments on different benchmarks and parameter analysis demonstrate the effectiveness and efficiency of MuSKPrompt in black-box few-shot text classification tasks.
Anthology ID:
2024.naacl-long.98
Volume:
Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)
Month:
June
Year:
2024
Address:
Mexico City, Mexico
Editors:
Kevin Duh, Helena Gomez, Steven Bethard
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1743–1757
Language:
URL:
https://aclanthology.org/2024.naacl-long.98
DOI:
Bibkey:
Cite (ACL):
Xiaojun Kuang, C. L. Philip Chen, Shuzhen Li, and Tong Zhang. 2024. Multi-Scale Prompt Memory-Augmented Model for Black-Box Scenarios. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 1743–1757, Mexico City, Mexico. Association for Computational Linguistics.
Cite (Informal):
Multi-Scale Prompt Memory-Augmented Model for Black-Box Scenarios (Kuang et al., NAACL 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.naacl-long.98.pdf
Copyright:
 2024.naacl-long.98.copyright.pdf