<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="yung-demberg-2025-crowdsourcing">
    <titleInfo>
        <title>On Crowdsourcing Task Design for Discourse Relation Annotation</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Frances</namePart>
        <namePart type="family">Yung</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Vera</namePart>
        <namePart type="family">Demberg</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2025-01</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of Context and Meaning: Navigating Disagreements in NLP Annotation</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Michael</namePart>
            <namePart type="family">Roth</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Dominik</namePart>
            <namePart type="family">Schlechtweg</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>International Committee on Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Abu Dhabi, UAE</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Interpreting implicit discourse relations involves complex reasoning, requiring the integration of semantic cues with background knowledge, as overt connectives like “because” or “then” are absent. These relations often allow multiple interpretations, best represented as distributions. In this study, we compare two established methods that crowdsource implicit discourse relation annotation by connective insertion: a free-choice approach, which allows annotators to select any suitable connective, and a forced-choice approach, which asks them to select among a set of predefined options. Specifically, we re-annotate the whole DiscoGeM 1.0 corpus - initially annotated with the free-choice method - using the forced-choice approach. The free-choice approach allows for flexible and intuitive insertion of various connectives, which are context-dependent. Comparison among over 130,000 annotations, however, shows that the free-choice strategy produces less diverse annotations, often converging on common labels. Analysis of the results reveals the interplay between task design and the annotators’ abilities to interpret and produce discourse relations.</abstract>
    <identifier type="citekey">yung-demberg-2025-crowdsourcing</identifier>
    <location>
        <url>https://aclanthology.org/2025.comedi-1.2/</url>
    </location>
    <part>
        <date>2025-01</date>
        <extent unit="page">
            <start>12</start>
            <end>19</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T On Crowdsourcing Task Design for Discourse Relation Annotation
%A Yung, Frances
%A Demberg, Vera
%Y Roth, Michael
%Y Schlechtweg, Dominik
%S Proceedings of Context and Meaning: Navigating Disagreements in NLP Annotation
%D 2025
%8 January
%I International Committee on Computational Linguistics
%C Abu Dhabi, UAE
%F yung-demberg-2025-crowdsourcing
%X Interpreting implicit discourse relations involves complex reasoning, requiring the integration of semantic cues with background knowledge, as overt connectives like “because” or “then” are absent. These relations often allow multiple interpretations, best represented as distributions. In this study, we compare two established methods that crowdsource implicit discourse relation annotation by connective insertion: a free-choice approach, which allows annotators to select any suitable connective, and a forced-choice approach, which asks them to select among a set of predefined options. Specifically, we re-annotate the whole DiscoGeM 1.0 corpus - initially annotated with the free-choice method - using the forced-choice approach. The free-choice approach allows for flexible and intuitive insertion of various connectives, which are context-dependent. Comparison among over 130,000 annotations, however, shows that the free-choice strategy produces less diverse annotations, often converging on common labels. Analysis of the results reveals the interplay between task design and the annotators’ abilities to interpret and produce discourse relations.
%U https://aclanthology.org/2025.comedi-1.2/
%P 12-19

Markdown (Informal)

[On Crowdsourcing Task Design for Discourse Relation Annotation](https://aclanthology.org/2025.comedi-1.2/) (Yung & Demberg, CoMeDi 2025)

• On Crowdsourcing Task Design for Discourse Relation Annotation (Yung & Demberg, CoMeDi 2025)

ACL

• Frances Yung and Vera Demberg. 2025. On Crowdsourcing Task Design for Discourse Relation Annotation. In Proceedings of Context and Meaning: Navigating Disagreements in NLP Annotation, pages 12–19, Abu Dhabi, UAE. International Committee on Computational Linguistics.