A Novel Workflow for Accurately and Efficiently Crowdsourcing Predicate Senses and Argument Labels

Youxuan Jiang, Huaiyu Zhu, Jonathan K. Kummerfeld, Yunyao Li, Walter Lasecki


Abstract
Resources for Semantic Role Labeling (SRL) are typically annotated by experts at great expense. Prior attempts to develop crowdsourcing methods have either had low accuracy or required substantial expert annotation. We propose a new multi-stage crowd workflow that substantially reduces expert involvement without sacrificing accuracy. In particular, we introduce a unique filter stage based on the key observation that crowd workers are able to almost perfectly filter out incorrect options for labels. Our three-stage workflow produces annotations with 95% accuracy for predicate labels and 93% for argument labels, which is comparable to expert agreement. Compared to prior work on crowdsourcing for SRL, we decrease expert effort by 4x, from 56% to 14% of cases. Our approach enables more scalable annotation of SRL, and could enable annotation of NLP tasks that have previously been considered too complex to effectively crowdsource.
Anthology ID:
2020.findings-emnlp.38
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2020
Month:
November
Year:
2020
Address:
Online
Editors:
Trevor Cohn, Yulan He, Yang Liu
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
415–421
Language:
URL:
https://aclanthology.org/2020.findings-emnlp.38
DOI:
10.18653/v1/2020.findings-emnlp.38
Bibkey:
Cite (ACL):
Youxuan Jiang, Huaiyu Zhu, Jonathan K. Kummerfeld, Yunyao Li, and Walter Lasecki. 2020. A Novel Workflow for Accurately and Efficiently Crowdsourcing Predicate Senses and Argument Labels. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 415–421, Online. Association for Computational Linguistics.
Cite (Informal):
A Novel Workflow for Accurately and Efficiently Crowdsourcing Predicate Senses and Argument Labels (Jiang et al., Findings 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.findings-emnlp.38.pdf
Optional supplementary material:
 2020.findings-emnlp.38.OptionalSupplementaryMaterial.zip
Code
 system-t/crowdsourcingsrl