<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="karpinska-etal-2021-perils">
    <titleInfo>
        <title>The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Marzena</namePart>
        <namePart type="family">Karpinska</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Nader</namePart>
        <namePart type="family">Akoury</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Mohit</namePart>
        <namePart type="family">Iyyer</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2021-11</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing</title>
        </titleInfo>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Online and Punta Cana, Dominican Republic</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Recent text generation research has increasingly focused on open-ended domains such as story and poetry generation. Because models built for such tasks are difficult to evaluate automatically, most researchers in the space justify their modeling choices by collecting crowdsourced human judgments of text quality (e.g., Likert scores of coherence or grammaticality) from Amazon Mechanical Turk (AMT). In this paper, we first conduct a survey of 45 open-ended text generation papers and find that the vast majority of them fail to report crucial details about their AMT tasks, hindering reproducibility. We then run a series of story evaluation experiments with both AMT workers and English teachers and discover that even with strict qualification filters, AMT workers (unlike teachers) fail to distinguish between model-generated text and human-generated references. We show that AMT worker judgments improve when they are shown model-generated output alongside human-generated references, which enables the workers to better calibrate their ratings. Finally, interviews with the English teachers provide deeper insights into the challenges of the evaluation process, particularly when rating model-generated text.</abstract>
    <identifier type="citekey">karpinska-etal-2021-perils</identifier>
    <identifier type="doi">10.18653/v1/2021.emnlp-main.97</identifier>
    <location>
        <url>https://aclanthology.org/2021.emnlp-main.97</url>
    </location>
    <part>
        <date>2021-11</date>
        <extent unit="page">
            <start>1265</start>
            <end>1285</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation
%A Karpinska, Marzena
%A Akoury, Nader
%A Iyyer, Mohit
%S Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
%D 2021
%8 November
%I Association for Computational Linguistics
%C Online and Punta Cana, Dominican Republic
%F karpinska-etal-2021-perils
%X Recent text generation research has increasingly focused on open-ended domains such as story and poetry generation. Because models built for such tasks are difficult to evaluate automatically, most researchers in the space justify their modeling choices by collecting crowdsourced human judgments of text quality (e.g., Likert scores of coherence or grammaticality) from Amazon Mechanical Turk (AMT). In this paper, we first conduct a survey of 45 open-ended text generation papers and find that the vast majority of them fail to report crucial details about their AMT tasks, hindering reproducibility. We then run a series of story evaluation experiments with both AMT workers and English teachers and discover that even with strict qualification filters, AMT workers (unlike teachers) fail to distinguish between model-generated text and human-generated references. We show that AMT worker judgments improve when they are shown model-generated output alongside human-generated references, which enables the workers to better calibrate their ratings. Finally, interviews with the English teachers provide deeper insights into the challenges of the evaluation process, particularly when rating model-generated text.
%R 10.18653/v1/2021.emnlp-main.97
%U https://aclanthology.org/2021.emnlp-main.97
%U https://doi.org/10.18653/v1/2021.emnlp-main.97
%P 1265-1285

Markdown (Informal)

[The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation](https://aclanthology.org/2021.emnlp-main.97) (Karpinska et al., EMNLP 2021)

• The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation (Karpinska et al., EMNLP 2021)

ACL

• Marzena Karpinska, Nader Akoury, and Mohit Iyyer. 2021. The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1265–1285, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.