<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="gomes-etal-2020-effort">
    <titleInfo>
        <title>Effort Estimation in Named Entity Tagging Tasks</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Inês</namePart>
        <namePart type="family">Gomes</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Rui</namePart>
        <namePart type="family">Correia</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Jorge</namePart>
        <namePart type="family">Ribeiro</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">João</namePart>
        <namePart type="family">Freitas</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2020-05</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <language>
        <languageTerm type="text">English</languageTerm>
        <languageTerm type="code" authority="iso639-2b">eng</languageTerm>
    </language>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the Twelfth Language Resources and Evaluation Conference</title>
        </titleInfo>
        <originInfo>
            <publisher>European Language Resources Association</publisher>
            <place>
                <placeTerm type="text">Marseille, France</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-10-95546-34-4</identifier>
    </relatedItem>
    <abstract>Named Entity Recognition (NER) is an essential component of many Natural Language Processing pipelines. However, building these language dependent models requires large amounts of annotated data. Crowdsourcing emerged as a scalable solution to collect and enrich data in a more time-efficient manner. To manage these annotations at scale, it is important to predict completion timelines and compute fair pricing for workers in advance. To achieve these goals, we need to know how much effort will be taken to complete each task. In this paper, we investigate which variables influence the time spent on a named entity annotation task by a human. Our results are two-fold: first, the understanding of the effort-impacting factors which we divided into cognitive load and input length; and second, the performance of the prediction itself. On the latter, through model adaptation and feature engineering, we attained a Root Mean Squared Error (RMSE) of 25.68 words per minute with a Nearest Neighbors model.</abstract>
    <identifier type="citekey">gomes-etal-2020-effort</identifier>
    <location>
        <url>https://aclanthology.org/2020.lrec-1.37</url>
    </location>
    <part>
        <date>2020-05</date>
        <extent unit="page">
            <start>298</start>
            <end>306</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Effort Estimation in Named Entity Tagging Tasks
%A Gomes, Inês
%A Correia, Rui
%A Ribeiro, Jorge
%A Freitas, João
%S Proceedings of the Twelfth Language Resources and Evaluation Conference
%D 2020
%8 May
%I European Language Resources Association
%C Marseille, France
%@ 979-10-95546-34-4
%G English
%F gomes-etal-2020-effort
%X Named Entity Recognition (NER) is an essential component of many Natural Language Processing pipelines. However, building these language dependent models requires large amounts of annotated data. Crowdsourcing emerged as a scalable solution to collect and enrich data in a more time-efficient manner. To manage these annotations at scale, it is important to predict completion timelines and compute fair pricing for workers in advance. To achieve these goals, we need to know how much effort will be taken to complete each task. In this paper, we investigate which variables influence the time spent on a named entity annotation task by a human. Our results are two-fold: first, the understanding of the effort-impacting factors which we divided into cognitive load and input length; and second, the performance of the prediction itself. On the latter, through model adaptation and feature engineering, we attained a Root Mean Squared Error (RMSE) of 25.68 words per minute with a Nearest Neighbors model.
%U https://aclanthology.org/2020.lrec-1.37
%P 298-306

Markdown (Informal)

[Effort Estimation in Named Entity Tagging Tasks](https://aclanthology.org/2020.lrec-1.37) (Gomes et al., LREC 2020)

• Effort Estimation in Named Entity Tagging Tasks (Gomes et al., LREC 2020)

ACL

• Inês Gomes, Rui Correia, Jorge Ribeiro, and João Freitas. 2020. Effort Estimation in Named Entity Tagging Tasks. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 298–306, Marseille, France. European Language Resources Association.