<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="klie-etal-2023-annotation">
    <titleInfo>
        <title>Annotation Error Detection: Analyzing the Past and Present for a More Coherent Future</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Jan-Christoph</namePart>
        <namePart type="family">Klie</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Bonnie</namePart>
        <namePart type="family">Webber</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Iryna</namePart>
        <namePart type="family">Gurevych</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-03</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <genre authority="bibutilsgt">journal article</genre>
    <relatedItem type="host">
        <titleInfo>
            <title>Computational Linguistics</title>
        </titleInfo>
        <originInfo>
            <issuance>continuing</issuance>
            <publisher>MIT Press</publisher>
            <place>
                <placeTerm type="text">Cambridge, MA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">periodical</genre>
        <genre authority="bibutilsgt">academic journal</genre>
    </relatedItem>
    <abstract>Annotated data is an essential ingredient in natural language processing for training and evaluating machine learning models. It is therefore very desirable for the annotations to be of high quality. Recent work, however, has shown that several popular datasets contain a surprising number of annotation errors or inconsistencies. To alleviate this issue, many methods for annotation error detection have been devised over the years. While researchers show that their approaches work well on their newly introduced datasets, they rarely compare their methods to previous work or on the same datasets. This raises strong concerns on methods’ general performance and makes it difficult to assess their strengths and weaknesses. We therefore reimplement 18 methods for detecting potential annotation errors and evaluate them on 9 English datasets for text classification as well as token and span labeling. In addition, we define a uniform evaluation setup including a new formalization of the annotation error detection task, evaluation protocol, and general best practices. To facilitate future research and reproducibility, we release our datasets and implementations in an easy-to-use and open source software package.1</abstract>
    <identifier type="citekey">klie-etal-2023-annotation</identifier>
    <identifier type="doi">10.1162/coli_a_00464</identifier>
    <location>
        <url>https://aclanthology.org/2023.cl-1.4</url>
    </location>
    <part>
        <date>2023-03</date>
        <detail type="volume"><number>49</number></detail>
        <detail type="issue"><number>1</number></detail>
        <extent unit="page">
            <start>157</start>
            <end>198</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Journal Article
%T Annotation Error Detection: Analyzing the Past and Present for a More Coherent Future
%A Klie, Jan-Christoph
%A Webber, Bonnie
%A Gurevych, Iryna
%J Computational Linguistics
%D 2023
%8 March
%V 49
%N 1
%I MIT Press
%C Cambridge, MA
%F klie-etal-2023-annotation
%X Annotated data is an essential ingredient in natural language processing for training and evaluating machine learning models. It is therefore very desirable for the annotations to be of high quality. Recent work, however, has shown that several popular datasets contain a surprising number of annotation errors or inconsistencies. To alleviate this issue, many methods for annotation error detection have been devised over the years. While researchers show that their approaches work well on their newly introduced datasets, they rarely compare their methods to previous work or on the same datasets. This raises strong concerns on methods’ general performance and makes it difficult to assess their strengths and weaknesses. We therefore reimplement 18 methods for detecting potential annotation errors and evaluate them on 9 English datasets for text classification as well as token and span labeling. In addition, we define a uniform evaluation setup including a new formalization of the annotation error detection task, evaluation protocol, and general best practices. To facilitate future research and reproducibility, we release our datasets and implementations in an easy-to-use and open source software package.1
%R 10.1162/coli_a_00464
%U https://aclanthology.org/2023.cl-1.4
%U https://doi.org/10.1162/coli_a_00464
%P 157-198

Markdown (Informal)

[Annotation Error Detection: Analyzing the Past and Present for a More Coherent Future](https://aclanthology.org/2023.cl-1.4) (Klie et al., CL 2023)

• Annotation Error Detection: Analyzing the Past and Present for a More Coherent Future (Klie et al., CL 2023)

ACL

• Jan-Christoph Klie, Bonnie Webber, and Iryna Gurevych. 2023. Annotation Error Detection: Analyzing the Past and Present for a More Coherent Future. Computational Linguistics, 49(1):157–198.