DReCa: A General Task Augmentation Strategy for Few-Shot Natural Language Inference

Shikhar Murty, Tatsunori B. Hashimoto, Christopher Manning


Abstract
Meta-learning promises few-shot learners that can adapt to new distributions by repurposing knowledge acquired from previous training. However, we believe meta-learning has not yet succeeded in NLP due to the lack of a well-defined task distribution, leading to attempts that treat datasets as tasks. Such an ad hoc task distribution causes problems of quantity and quality. Since there’s only a handful of datasets for any NLP problem, meta-learners tend to overfit their adaptation mechanism and, since NLP datasets are highly heterogeneous, many learning episodes have poor transfer between their support and query sets, which discourages the meta-learner from adapting. To alleviate these issues, we propose DReCA (Decomposing datasets into Reasoning Categories), a simple method for discovering and using latent reasoning categories in a dataset, to form additional high quality tasks. DReCA works by splitting examples into label groups, embedding them with a finetuned BERT model and then clustering each group into reasoning categories. Across four few-shot NLI problems, we demonstrate that using DReCA improves the accuracy of meta-learners by 1.5-4%
Anthology ID:
2021.naacl-main.88
Volume:
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Month:
June
Year:
2021
Address:
Online
Editors:
Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, Yichao Zhou
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1113–1125
Language:
URL:
https://aclanthology.org/2021.naacl-main.88
DOI:
10.18653/v1/2021.naacl-main.88
Bibkey:
Cite (ACL):
Shikhar Murty, Tatsunori B. Hashimoto, and Christopher Manning. 2021. DReCa: A General Task Augmentation Strategy for Few-Shot Natural Language Inference. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1113–1125, Online. Association for Computational Linguistics.
Cite (Informal):
DReCa: A General Task Augmentation Strategy for Few-Shot Natural Language Inference (Murty et al., NAACL 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.naacl-main.88.pdf
Video:
 https://aclanthology.org/2021.naacl-main.88.mp4
Data
GLUEMultiNLI