<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="nouri-etal-2020-mining">
    <titleInfo>
        <title>Mining Crowdsourcing Problems from Discussion Forums of Workers</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Zahra</namePart>
        <namePart type="family">Nouri</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Henning</namePart>
        <namePart type="family">Wachsmuth</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Gregor</namePart>
        <namePart type="family">Engels</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2020-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 28th International Conference on Computational Linguistics</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Donia</namePart>
            <namePart type="family">Scott</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Nuria</namePart>
            <namePart type="family">Bel</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Chengqing</namePart>
            <namePart type="family">Zong</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>International Committee on Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Barcelona, Spain (Online)</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Crowdsourcing is used in academia and industry to solve tasks that are easy for humans but hard for computers, in natural language processing mostly to annotate data. The quality of annotations is affected by problems in the task design, task operation, and task evaluation that workers face with requesters in crowdsourcing processes. To learn about the major problems, we provide a short but comprehensive survey based on two complementary studies: (1) a literature review where we collect and organize problems known from interviews with workers, and (2) an empirical data analysis where we use topic modeling to mine workers’ complaints from a new English corpus of workers’ forum discussions. While literature covers all process phases, problems in the task evaluation are prevalent, including unfair rejections, late payments, and unjustified blockings of workers. According to the data, however, poor task design in terms of malfunctioning environments, bad workload estimation, and privacy violations seems to bother the workers most. Our findings form the basis for future research on how to improve crowdsourcing processes.</abstract>
    <identifier type="citekey">nouri-etal-2020-mining</identifier>
    <identifier type="doi">10.18653/v1/2020.coling-main.551</identifier>
    <location>
        <url>https://aclanthology.org/2020.coling-main.551</url>
    </location>
    <part>
        <date>2020-12</date>
        <extent unit="page">
            <start>6264</start>
            <end>6276</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Mining Crowdsourcing Problems from Discussion Forums of Workers
%A Nouri, Zahra
%A Wachsmuth, Henning
%A Engels, Gregor
%Y Scott, Donia
%Y Bel, Nuria
%Y Zong, Chengqing
%S Proceedings of the 28th International Conference on Computational Linguistics
%D 2020
%8 December
%I International Committee on Computational Linguistics
%C Barcelona, Spain (Online)
%F nouri-etal-2020-mining
%X Crowdsourcing is used in academia and industry to solve tasks that are easy for humans but hard for computers, in natural language processing mostly to annotate data. The quality of annotations is affected by problems in the task design, task operation, and task evaluation that workers face with requesters in crowdsourcing processes. To learn about the major problems, we provide a short but comprehensive survey based on two complementary studies: (1) a literature review where we collect and organize problems known from interviews with workers, and (2) an empirical data analysis where we use topic modeling to mine workers’ complaints from a new English corpus of workers’ forum discussions. While literature covers all process phases, problems in the task evaluation are prevalent, including unfair rejections, late payments, and unjustified blockings of workers. According to the data, however, poor task design in terms of malfunctioning environments, bad workload estimation, and privacy violations seems to bother the workers most. Our findings form the basis for future research on how to improve crowdsourcing processes.
%R 10.18653/v1/2020.coling-main.551
%U https://aclanthology.org/2020.coling-main.551
%U https://doi.org/10.18653/v1/2020.coling-main.551
%P 6264-6276

Markdown (Informal)

[Mining Crowdsourcing Problems from Discussion Forums of Workers](https://aclanthology.org/2020.coling-main.551) (Nouri et al., COLING 2020)

• Mining Crowdsourcing Problems from Discussion Forums of Workers (Nouri et al., COLING 2020)

ACL

• Zahra Nouri, Henning Wachsmuth, and Gregor Engels. 2020. Mining Crowdsourcing Problems from Discussion Forums of Workers. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6264–6276, Barcelona, Spain (Online). International Committee on Computational Linguistics.