ConVEx: Data-Efficient and Few-Shot Slot Labeling

Matthew Henderson, Ivan Vulić


Abstract
We propose ConVEx (Conversational Value Extractor), an efficient pretraining and fine-tuning neural approach for slot-labeling dialog tasks. Instead of relying on more general pretraining objectives from prior work (e.g., language modeling, response selection), ConVEx’s pretraining objective, a novel pairwise cloze task using Reddit data, is well aligned with its intended usage on sequence labeling tasks. This enables learning domain-specific slot labelers by simply fine-tuning decoding layers of the pretrained general-purpose sequence labeling model, while the majority of the pretrained model’s parameters are kept frozen. We report state-of-the-art performance of ConVEx across a range of diverse domains and data sets for dialog slot-labeling, with the largest gains in the most challenging, few-shot setups. We believe that ConVEx’s reduced pretraining times (i.e., only 18 hours on 12 GPUs) and cost, along with its efficient fine-tuning and strong performance, promise wider portability and scalability for data-efficient sequence-labeling tasks in general.
Anthology ID:
2021.naacl-main.264
Volume:
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Month:
June
Year:
2021
Address:
Online
Editors:
Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, Yichao Zhou
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
3375–3389
Language:
URL:
https://aclanthology.org/2021.naacl-main.264
DOI:
10.18653/v1/2021.naacl-main.264
Bibkey:
Cite (ACL):
Matthew Henderson and Ivan Vulić. 2021. ConVEx: Data-Efficient and Few-Shot Slot Labeling. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3375–3389, Online. Association for Computational Linguistics.
Cite (Informal):
ConVEx: Data-Efficient and Few-Shot Slot Labeling (Henderson & Vulić, NAACL 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.naacl-main.264.pdf
Video:
 https://aclanthology.org/2021.naacl-main.264.mp4
Data
CC100SNIPS