Inconsistencies in Crowdsourced Slot-Filling Annotations: A Typology and Identification Methods

Stefan Larson, Adrian Cheung, Anish Mahendran, Kevin Leach, Jonathan K. Kummerfeld


Abstract
Slot-filling models in task-driven dialog systems rely on carefully annotated training data. However, annotations by crowd workers are often inconsistent or contain errors. Simple solutions like manually checking annotations or having multiple workers label each sample are expensive and waste effort on samples that are correct. If we can identify inconsistencies, we can focus effort where it is needed. Toward this end, we define six inconsistency types in slot-filling annotations. Using three new noisy crowd-annotated datasets, we show that a wide range of inconsistencies occur and can impact system performance if not addressed. We then introduce automatic methods of identifying inconsistencies. Experiments on our new datasets show that these methods effectively reveal inconsistencies in data, though there is further scope for improvement.
Anthology ID:
2020.coling-main.442
Volume:
Proceedings of the 28th International Conference on Computational Linguistics
Month:
December
Year:
2020
Address:
Barcelona, Spain (Online)
Editors:
Donia Scott, Nuria Bel, Chengqing Zong
Venue:
COLING
SIG:
Publisher:
International Committee on Computational Linguistics
Note:
Pages:
5035–5046
Language:
URL:
https://aclanthology.org/2020.coling-main.442
DOI:
10.18653/v1/2020.coling-main.442
Bibkey:
Cite (ACL):
Stefan Larson, Adrian Cheung, Anish Mahendran, Kevin Leach, and Jonathan K. Kummerfeld. 2020. Inconsistencies in Crowdsourced Slot-Filling Annotations: A Typology and Identification Methods. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5035–5046, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Cite (Informal):
Inconsistencies in Crowdsourced Slot-Filling Annotations: A Typology and Identification Methods (Larson et al., COLING 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.coling-main.442.pdf