Mining Crowdsourcing Problems from Discussion Forums of Workers

Zahra Nouri, Henning Wachsmuth, Gregor Engels


Abstract
Crowdsourcing is used in academia and industry to solve tasks that are easy for humans but hard for computers, in natural language processing mostly to annotate data. The quality of annotations is affected by problems in the task design, task operation, and task evaluation that workers face with requesters in crowdsourcing processes. To learn about the major problems, we provide a short but comprehensive survey based on two complementary studies: (1) a literature review where we collect and organize problems known from interviews with workers, and (2) an empirical data analysis where we use topic modeling to mine workers’ complaints from a new English corpus of workers’ forum discussions. While literature covers all process phases, problems in the task evaluation are prevalent, including unfair rejections, late payments, and unjustified blockings of workers. According to the data, however, poor task design in terms of malfunctioning environments, bad workload estimation, and privacy violations seems to bother the workers most. Our findings form the basis for future research on how to improve crowdsourcing processes.
Anthology ID:
2020.coling-main.551
Volume:
Proceedings of the 28th International Conference on Computational Linguistics
Month:
December
Year:
2020
Address:
Barcelona, Spain (Online)
Editors:
Donia Scott, Nuria Bel, Chengqing Zong
Venue:
COLING
SIG:
Publisher:
International Committee on Computational Linguistics
Note:
Pages:
6264–6276
Language:
URL:
https://aclanthology.org/2020.coling-main.551
DOI:
10.18653/v1/2020.coling-main.551
Bibkey:
Cite (ACL):
Zahra Nouri, Henning Wachsmuth, and Gregor Engels. 2020. Mining Crowdsourcing Problems from Discussion Forums of Workers. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6264–6276, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Cite (Informal):
Mining Crowdsourcing Problems from Discussion Forums of Workers (Nouri et al., COLING 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.coling-main.551.pdf