Label Semantic Aware Pre-training for Few-shot Text Classification

Aaron Mueller, Jason Krone, Salvatore Romeo, Saab Mansour, Elman Mansimov, Yi Zhang, Dan Roth


Abstract
In text classification tasks, useful information is encoded in the label names. Label semantic aware systems have leveraged this information for improved text classification performance during fine-tuning and prediction. However, use of label-semantics during pre-training has not been extensively explored. We therefore propose Label Semantic Aware Pre-training (LSAP) to improve the generalization and data efficiency of text classification systems. LSAP incorporates label semantics into pre-trained generative models (T5 in our case) by performing secondary pre-training on labeled sentences from a variety of domains. As domain-general pre-training requires large amounts of data, we develop a filtering and labeling pipeline to automatically create sentence-label pairs from unlabeled text. We perform experiments on intent (ATIS, Snips, TOPv2) and topic classification (AG News, Yahoo! Answers). LSAP obtains significant accuracy improvements over state-of-the-art models for few-shot text classification while maintaining performance comparable to state of the art in high-resource settings.
Anthology ID:
2022.acl-long.570
Volume:
Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Month:
May
Year:
2022
Address:
Dublin, Ireland
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
8318–8334
Language:
URL:
https://aclanthology.org/2022.acl-long.570
DOI:
10.18653/v1/2022.acl-long.570
Bibkey:
Cite (ACL):
Aaron Mueller, Jason Krone, Salvatore Romeo, Saab Mansour, Elman Mansimov, Yi Zhang, and Dan Roth. 2022. Label Semantic Aware Pre-training for Few-shot Text Classification. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8318–8334, Dublin, Ireland. Association for Computational Linguistics.
Cite (Informal):
Label Semantic Aware Pre-training for Few-shot Text Classification (Mueller et al., ACL 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.acl-long.570.pdf
Video:
 https://aclanthology.org/2022.acl-long.570.mp4
Code
 amazon-research/label-aware-pretrain
Data
SGDSNIPSTOPv2