<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="felt-etal-2018-learning">
    <titleInfo>
        <title>Learning from Measurements in Crowdsourcing Models: Inferring Ground Truth from Diverse Annotation Types</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Paul</namePart>
        <namePart type="family">Felt</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Eric</namePart>
        <namePart type="family">Ringger</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Jordan</namePart>
        <namePart type="family">Boyd-Graber</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Kevin</namePart>
        <namePart type="family">Seppi</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2018-08</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 27th International Conference on Computational Linguistics</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Emily</namePart>
            <namePart type="given">M</namePart>
            <namePart type="family">Bender</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Leon</namePart>
            <namePart type="family">Derczynski</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Pierre</namePart>
            <namePart type="family">Isabelle</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Santa Fe, New Mexico, USA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Annotated corpora enable supervised machine learning and data analysis. To reduce the cost of manual annotation, tasks are often assigned to internet workers whose judgments are reconciled by crowdsourcing models. We approach the problem of crowdsourcing using a framework for learning from rich prior knowledge, and we identify a family of crowdsourcing models with the novel ability to combine annotations with differing structures: e.g., document labels and word labels. Annotator judgments are given in the form of the predicted expected value of measurement functions computed over annotations and the data, unifying annotation models. Our model, a specific instance of this framework, compares favorably with previous work. Furthermore, it enables active sample selection, jointly selecting annotator, data item, and annotation structure to reduce annotation effort.</abstract>
    <identifier type="citekey">felt-etal-2018-learning</identifier>
    <location>
        <url>https://aclanthology.org/C18-1144</url>
    </location>
    <part>
        <date>2018-08</date>
        <extent unit="page">
            <start>1694</start>
            <end>1704</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Learning from Measurements in Crowdsourcing Models: Inferring Ground Truth from Diverse Annotation Types
%A Felt, Paul
%A Ringger, Eric
%A Boyd-Graber, Jordan
%A Seppi, Kevin
%Y Bender, Emily M.
%Y Derczynski, Leon
%Y Isabelle, Pierre
%S Proceedings of the 27th International Conference on Computational Linguistics
%D 2018
%8 August
%I Association for Computational Linguistics
%C Santa Fe, New Mexico, USA
%F felt-etal-2018-learning
%X Annotated corpora enable supervised machine learning and data analysis. To reduce the cost of manual annotation, tasks are often assigned to internet workers whose judgments are reconciled by crowdsourcing models. We approach the problem of crowdsourcing using a framework for learning from rich prior knowledge, and we identify a family of crowdsourcing models with the novel ability to combine annotations with differing structures: e.g., document labels and word labels. Annotator judgments are given in the form of the predicted expected value of measurement functions computed over annotations and the data, unifying annotation models. Our model, a specific instance of this framework, compares favorably with previous work. Furthermore, it enables active sample selection, jointly selecting annotator, data item, and annotation structure to reduce annotation effort.
%U https://aclanthology.org/C18-1144
%P 1694-1704

Markdown (Informal)

[Learning from Measurements in Crowdsourcing Models: Inferring Ground Truth from Diverse Annotation Types](https://aclanthology.org/C18-1144) (Felt et al., COLING 2018)

• Learning from Measurements in Crowdsourcing Models: Inferring Ground Truth from Diverse Annotation Types (Felt et al., COLING 2018)

ACL

• Paul Felt, Eric Ringger, Jordan Boyd-Graber, and Kevin Seppi. 2018. Learning from Measurements in Crowdsourcing Models: Inferring Ground Truth from Diverse Annotation Types. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1694–1704, Santa Fe, New Mexico, USA. Association for Computational Linguistics.