LM-BFF-MS: Improving Few-Shot Fine-tuning of Language Models based on Multiple Soft Demonstration Memory

Eunhwan Park, Donghyeon Jeon, Seonhoon Kim, Inho Kang, Seung-Hoon Na


Abstract
LM-BFF (CITATION) achieves significant few-shot performance by using auto-generated prompts and adding demonstrations similar to an input example. To improve the approach of LM-BFF, this paper proposes LM-BFF-MSbetter few-shot fine-tuning of language models with multiple soft demonstrations by making its further extensions, which include 1) prompts with multiple demonstrations based on automatic generation of multiple label words; and 2) soft demonstration memory which consists of multiple sequences of globally shared word embeddings for a similar context. Experiments conducted on eight NLP tasks show that LM-BFF-MS leads to improvements over LM-BFF on five tasks, particularly achieving 94.0 and 90.4 on SST-2 and MRPC, respectively.
Anthology ID:
2022.acl-short.34
Volume:
Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
Month:
May
Year:
2022
Address:
Dublin, Ireland
Editors:
Smaranda Muresan, Preslav Nakov, Aline Villavicencio
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
310–317
Language:
URL:
https://aclanthology.org/2022.acl-short.34
DOI:
10.18653/v1/2022.acl-short.34
Bibkey:
Cite (ACL):
Eunhwan Park, Donghyeon Jeon, Seonhoon Kim, Inho Kang, and Seung-Hoon Na. 2022. LM-BFF-MS: Improving Few-Shot Fine-tuning of Language Models based on Multiple Soft Demonstration Memory. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 310–317, Dublin, Ireland. Association for Computational Linguistics.
Cite (Informal):
LM-BFF-MS: Improving Few-Shot Fine-tuning of Language Models based on Multiple Soft Demonstration Memory (Park et al., ACL 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.acl-short.34.pdf
Software:
 2022.acl-short.34.software.tgz
Code
 judepark96/lm-bff-ms
Data
MPQA Opinion CorpusMRPCMultiNLISNLISSTSST-2