<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="tratz-2019-dependency">
    <titleInfo>
        <title>Dependency Tree Annotation with Mechanical Turk</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Stephen</namePart>
        <namePart type="family">Tratz</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2019-11</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the First Workshop on Aggregating and Analysing Crowdsourced Annotations for NLP</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Silviu</namePart>
            <namePart type="family">Paun</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Dirk</namePart>
            <namePart type="family">Hovy</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Hong Kong</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Crowdsourcing is frequently employed to quickly and inexpensively obtain valuable linguistic annotations but is rarely used for parsing, likely due to the perceived difficulty of the task and the limited training of the available workers. This paper presents what is, to the best of our knowledge, the first published use of Mechanical Turk (or similar platform) to crowdsource parse trees. We pay Turkers to construct unlabeled dependency trees for 500 English sentences using an interactive graphical dependency tree editor, collecting 10 annotations per sentence. Despite not requiring any training, several of the more prolific workers meet or exceed 90% attachment agreement with the Penn Treebank (PTB) portion of our data, and, furthermore, for 72% of these PTB sentences, at least one Turker produces a perfect parse. Thus, we find that, supported with a simple graphical interface, people with presumably no prior experience can achieve surprisingly high degrees of accuracy on this task. To facilitate research into aggregation techniques for complex crowdsourced annotations, we publicly release our annotated corpus.</abstract>
    <identifier type="citekey">tratz-2019-dependency</identifier>
    <identifier type="doi">10.18653/v1/D19-5901</identifier>
    <location>
        <url>https://aclanthology.org/D19-5901</url>
    </location>
    <part>
        <date>2019-11</date>
        <extent unit="page">
            <start>1</start>
            <end>5</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Dependency Tree Annotation with Mechanical Turk
%A Tratz, Stephen
%Y Paun, Silviu
%Y Hovy, Dirk
%S Proceedings of the First Workshop on Aggregating and Analysing Crowdsourced Annotations for NLP
%D 2019
%8 November
%I Association for Computational Linguistics
%C Hong Kong
%F tratz-2019-dependency
%X Crowdsourcing is frequently employed to quickly and inexpensively obtain valuable linguistic annotations but is rarely used for parsing, likely due to the perceived difficulty of the task and the limited training of the available workers. This paper presents what is, to the best of our knowledge, the first published use of Mechanical Turk (or similar platform) to crowdsource parse trees. We pay Turkers to construct unlabeled dependency trees for 500 English sentences using an interactive graphical dependency tree editor, collecting 10 annotations per sentence. Despite not requiring any training, several of the more prolific workers meet or exceed 90% attachment agreement with the Penn Treebank (PTB) portion of our data, and, furthermore, for 72% of these PTB sentences, at least one Turker produces a perfect parse. Thus, we find that, supported with a simple graphical interface, people with presumably no prior experience can achieve surprisingly high degrees of accuracy on this task. To facilitate research into aggregation techniques for complex crowdsourced annotations, we publicly release our annotated corpus.
%R 10.18653/v1/D19-5901
%U https://aclanthology.org/D19-5901
%U https://doi.org/10.18653/v1/D19-5901
%P 1-5

Markdown (Informal)

[Dependency Tree Annotation with Mechanical Turk](https://aclanthology.org/D19-5901) (Tratz, 2019)

• Dependency Tree Annotation with Mechanical Turk (Tratz, 2019)

ACL

• Stephen Tratz. 2019. Dependency Tree Annotation with Mechanical Turk. In Proceedings of the First Workshop on Aggregating and Analysing Crowdsourced Annotations for NLP, pages 1–5, Hong Kong. Association for Computational Linguistics.