<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="thorn-jakobsen-etal-2022-sensitivity">
    <titleInfo>
        <title>The Sensitivity of Annotator Bias to Task Definitions in Argument Mining</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Terne</namePart>
        <namePart type="given">Sasha</namePart>
        <namePart type="family">Thorn Jakobsen</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Maria</namePart>
        <namePart type="family">Barrett</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Anders</namePart>
        <namePart type="family">Søgaard</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">David</namePart>
        <namePart type="family">Lassen</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2022-06</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 16th Linguistic Annotation Workshop (LAW-XVI) within LREC2022</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Sameer</namePart>
            <namePart type="family">Pradhan</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Sandra</namePart>
            <namePart type="family">Kuebler</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>European Language Resources Association</publisher>
            <place>
                <placeTerm type="text">Marseille, France</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>NLP models are dependent on the data they are trained on, including how this data is annotated. NLP research increasingly examines the social biases of models, but often in the light of their training data and specific social biases that can be identified in the text itself. In this paper, we present an annotation experiment that is the first to examine the extent to which social bias is sensitive to how data is annotated. We do so by collecting annotations of arguments in the same documents following four different guidelines and from four different demographic annotator backgrounds. We show that annotations exhibit widely different levels of group disparity depending on which guidelines annotators follow. The differences are not explained by task complexity, but rather by characteristics of these demographic groups, as previously identified by sociological studies. We release a dataset that is small in the number of instances but large in the number of annotations with demographic information, and our results encourage an increased awareness of annotator bias.</abstract>
    <identifier type="citekey">thorn-jakobsen-etal-2022-sensitivity</identifier>
    <location>
        <url>https://aclanthology.org/2022.law-1.6</url>
    </location>
    <part>
        <date>2022-06</date>
        <extent unit="page">
            <start>44</start>
            <end>61</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T The Sensitivity of Annotator Bias to Task Definitions in Argument Mining
%A Thorn Jakobsen, Terne Sasha
%A Barrett, Maria
%A Søgaard, Anders
%A Lassen, David
%Y Pradhan, Sameer
%Y Kuebler, Sandra
%S Proceedings of the 16th Linguistic Annotation Workshop (LAW-XVI) within LREC2022
%D 2022
%8 June
%I European Language Resources Association
%C Marseille, France
%F thorn-jakobsen-etal-2022-sensitivity
%X NLP models are dependent on the data they are trained on, including how this data is annotated. NLP research increasingly examines the social biases of models, but often in the light of their training data and specific social biases that can be identified in the text itself. In this paper, we present an annotation experiment that is the first to examine the extent to which social bias is sensitive to how data is annotated. We do so by collecting annotations of arguments in the same documents following four different guidelines and from four different demographic annotator backgrounds. We show that annotations exhibit widely different levels of group disparity depending on which guidelines annotators follow. The differences are not explained by task complexity, but rather by characteristics of these demographic groups, as previously identified by sociological studies. We release a dataset that is small in the number of instances but large in the number of annotations with demographic information, and our results encourage an increased awareness of annotator bias.
%U https://aclanthology.org/2022.law-1.6
%P 44-61

Markdown (Informal)

[The Sensitivity of Annotator Bias to Task Definitions in Argument Mining](https://aclanthology.org/2022.law-1.6) (Thorn Jakobsen et al., LAW 2022)

• The Sensitivity of Annotator Bias to Task Definitions in Argument Mining (Thorn Jakobsen et al., LAW 2022)

ACL

• Terne Sasha Thorn Jakobsen, Maria Barrett, Anders Søgaard, and David Lassen. 2022. The Sensitivity of Annotator Bias to Task Definitions in Argument Mining. In Proceedings of the 16th Linguistic Annotation Workshop (LAW-XVI) within LREC2022, pages 44–61, Marseille, France. European Language Resources Association.