A Two-stage Model for Slot Filling in Low-resource Settings: Domain-agnostic Non-slot Reduction and Pretrained Contextual Embeddings

Cennet Oguz, Ngoc Thang Vu


Abstract
Learning-based slot filling - a key component of spoken language understanding systems - typically requires a large amount of in-domain hand-labeled data for training. In this paper, we propose a novel two-stage model architecture that can be trained with only a few in-domain hand-labeled examples. The first step is designed to remove non-slot tokens (i.e., O labeled tokens), as they introduce noise in the input of slot filling models. This step is domain-agnostic and therefore, can be trained by exploiting out-of-domain data. The second step identifies slot names only for slot tokens by using state-of-the-art pretrained contextual embeddings such as ELMO and BERT. We show that our approach outperforms other state-of-art systems on the SNIPS benchmark dataset.
Anthology ID:
2020.sustainlp-1.10
Volume:
Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing
Month:
November
Year:
2020
Address:
Online
Editors:
Nafise Sadat Moosavi, Angela Fan, Vered Shwartz, Goran Glavaš, Shafiq Joty, Alex Wang, Thomas Wolf
Venue:
sustainlp
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
73–82
Language:
URL:
https://aclanthology.org/2020.sustainlp-1.10
DOI:
10.18653/v1/2020.sustainlp-1.10
Bibkey:
Cite (ACL):
Cennet Oguz and Ngoc Thang Vu. 2020. A Two-stage Model for Slot Filling in Low-resource Settings: Domain-agnostic Non-slot Reduction and Pretrained Contextual Embeddings. In Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing, pages 73–82, Online. Association for Computational Linguistics.
Cite (Informal):
A Two-stage Model for Slot Filling in Low-resource Settings: Domain-agnostic Non-slot Reduction and Pretrained Contextual Embeddings (Oguz & Vu, sustainlp 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.sustainlp-1.10.pdf
Video:
 https://slideslive.com/38939432
Data
SNIPS