<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="weber-plank-2023-activeaed">
    <titleInfo>
        <title>ActiveAED: A Human in the Loop Improves Annotation Error Detection</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Leon</namePart>
        <namePart type="family">Weber</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Barbara</namePart>
        <namePart type="family">Plank</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-07</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: ACL 2023</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Anna</namePart>
            <namePart type="family">Rogers</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jordan</namePart>
            <namePart type="family">Boyd-Graber</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Naoaki</namePart>
            <namePart type="family">Okazaki</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Toronto, Canada</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Manually annotated datasets are crucial for training and evaluating Natural Language Processing models. However, recent work has discovered that even widely-used benchmark datasets contain a substantial number of erroneous annotations. This problem has been addressed with Annotation Error Detection (AED) models, which can flag such errors for human re-annotation. However, even though many of these AED methods assume a final curation step in which a human annotator decides whether the annotation is erroneous, they have been developed as static models without any human-in-the-loop component. In this work, we propose ActiveAED, an AED method that can detect errors more accurately by repeatedly querying a human for error corrections in its prediction loop. We evaluate ActiveAED on eight datasets spanning five different tasks and find that it leads to improvements over the state of the art on seven of them, with gains of up to six percentage points in average precision.</abstract>
    <identifier type="citekey">weber-plank-2023-activeaed</identifier>
    <identifier type="doi">10.18653/v1/2023.findings-acl.562</identifier>
    <location>
        <url>https://aclanthology.org/2023.findings-acl.562</url>
    </location>
    <part>
        <date>2023-07</date>
        <extent unit="page">
            <start>8834</start>
            <end>8845</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T ActiveAED: A Human in the Loop Improves Annotation Error Detection
%A Weber, Leon
%A Plank, Barbara
%Y Rogers, Anna
%Y Boyd-Graber, Jordan
%Y Okazaki, Naoaki
%S Findings of the Association for Computational Linguistics: ACL 2023
%D 2023
%8 July
%I Association for Computational Linguistics
%C Toronto, Canada
%F weber-plank-2023-activeaed
%X Manually annotated datasets are crucial for training and evaluating Natural Language Processing models. However, recent work has discovered that even widely-used benchmark datasets contain a substantial number of erroneous annotations. This problem has been addressed with Annotation Error Detection (AED) models, which can flag such errors for human re-annotation. However, even though many of these AED methods assume a final curation step in which a human annotator decides whether the annotation is erroneous, they have been developed as static models without any human-in-the-loop component. In this work, we propose ActiveAED, an AED method that can detect errors more accurately by repeatedly querying a human for error corrections in its prediction loop. We evaluate ActiveAED on eight datasets spanning five different tasks and find that it leads to improvements over the state of the art on seven of them, with gains of up to six percentage points in average precision.
%R 10.18653/v1/2023.findings-acl.562
%U https://aclanthology.org/2023.findings-acl.562
%U https://doi.org/10.18653/v1/2023.findings-acl.562
%P 8834-8845

Markdown (Informal)

[ActiveAED: A Human in the Loop Improves Annotation Error Detection](https://aclanthology.org/2023.findings-acl.562) (Weber & Plank, Findings 2023)

• ActiveAED: A Human in the Loop Improves Annotation Error Detection (Weber & Plank, Findings 2023)

ACL

• Leon Weber and Barbara Plank. 2023. ActiveAED: A Human in the Loop Improves Annotation Error Detection. In Findings of the Association for Computational Linguistics: ACL 2023, pages 8834–8845, Toronto, Canada. Association for Computational Linguistics.