Know Where You’re Going: Meta-Learning for Parameter-Efficient Fine-Tuning

Mozhdeh Gheini, Xuezhe Ma, Jonathan May


Abstract
A recent family of techniques, dubbed lightweight fine-tuning methods, facilitates parameter-efficient transfer by updating only a small set of additional parameters while keeping the parameters of the original model frozen. While proven to be an effective approach, there are no existing studies on if and how such knowledge of the downstream fine-tuning approach calls for complementary measures after pre-training and before fine-tuning. In this work, we show that taking the ultimate choice of fine-tuning into consideration boosts the performance of parameter-efficient fine-tuning. By relying on optimization-based meta-learning using MAML with certain modifications for our distinct purpose, we prime the pre-trained model specifically for parameter-efficient fine-tuning, resulting in gains of up to 4.96 points on cross-lingual NER fine-tuning. Our ablation settings and analyses further reveal that the specific approach we take to meta-learning is crucial for the attained gains.
Anthology ID:
2023.findings-acl.737
Volume:
Findings of the Association for Computational Linguistics: ACL 2023
Month:
July
Year:
2023
Address:
Toronto, Canada
Editors:
Anna Rogers, Jordan Boyd-Graber, Naoaki Okazaki
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
11602–11612
Language:
URL:
https://aclanthology.org/2023.findings-acl.737
DOI:
10.18653/v1/2023.findings-acl.737
Bibkey:
Cite (ACL):
Mozhdeh Gheini, Xuezhe Ma, and Jonathan May. 2023. Know Where You’re Going: Meta-Learning for Parameter-Efficient Fine-Tuning. In Findings of the Association for Computational Linguistics: ACL 2023, pages 11602–11612, Toronto, Canada. Association for Computational Linguistics.
Cite (Informal):
Know Where You’re Going: Meta-Learning for Parameter-Efficient Fine-Tuning (Gheini et al., Findings 2023)
Copy Citation:
PDF:
https://aclanthology.org/2023.findings-acl.737.pdf