Few-Shot Learning with Siamese Networks and Label Tuning

Thomas Müller, Guillermo Pérez-Torró, Marc Franco-Salvador


Abstract
We study the problem of building text classifiers with little or no training data, commonly known as zero and few-shot text classification. In recent years, an approach based on neural textual entailment models has been found to give strong results on a diverse range of tasks. In this work, we show that with proper pre-training, Siamese Networks that embed texts and labels offer a competitive alternative. These models allow for a large reduction in inference cost: constant in the number of labels rather than linear. Furthermore, we introduce label tuning, a simple and computationally efficient approach that allows to adapt the models in a few-shot setup by only changing the label embeddings. While giving lower performance than model fine-tuning, this approach has the architectural advantage that a single encoder can be shared by many different tasks.
Anthology ID:
2022.acl-long.584
Volume:
Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Month:
May
Year:
2022
Address:
Dublin, Ireland
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
8532–8545
Language:
URL:
https://aclanthology.org/2022.acl-long.584
DOI:
10.18653/v1/2022.acl-long.584
Bibkey:
Cite (ACL):
Thomas Müller, Guillermo Pérez-Torró, and Marc Franco-Salvador. 2022. Few-Shot Learning with Siamese Networks and Label Tuning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8532–8545, Dublin, Ireland. Association for Computational Linguistics.
Cite (Informal):
Few-Shot Learning with Siamese Networks and Label Tuning (Müller et al., ACL 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.acl-long.584.pdf
Video:
 https://aclanthology.org/2022.acl-long.584.mp4
Code
 symanto-research/few-shot-learning-label-tuning
Data
AG NewsCoLAHeadQAIMDb Movie ReviewsISEARMultiNLISNLI