<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="ortmann-2022-fine">
    <titleInfo>
        <title>Fine-Grained Error Analysis and Fair Evaluation of Labeled Spans</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Katrin</namePart>
        <namePart type="family">Ortmann</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2022-06</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the Thirteenth Language Resources and Evaluation Conference</title>
        </titleInfo>
        <originInfo>
            <publisher>European Language Resources Association</publisher>
            <place>
                <placeTerm type="text">Marseille, France</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>The traditional evaluation of labeled spans with precision, recall, and F1-score has undesirable effects due to double penalties. Annotations with incorrect label or boundaries count as two errors instead of one, despite being closer to the target annotation than false positives or false negatives. In this paper, new error types are introduced, which more accurately reflect true annotation quality and ensure that every annotation counts only once. An algorithm for error identification in flat and multi-level annotations is presented and complemented with a proposal on how to calculate meaningful precision, recall, and F1-scores based on the more fine-grained error types. The exemplary application to three different annotation tasks (NER, chunking, parsing) shows that the suggested procedure not only prevents double penalties but also allows for a more detailed error analysis, thereby providing more insight into the actual weaknesses of a system.</abstract>
    <identifier type="citekey">ortmann-2022-fine</identifier>
    <location>
        <url>https://aclanthology.org/2022.lrec-1.150</url>
    </location>
    <part>
        <date>2022-06</date>
        <extent unit="page">
            <start>1400</start>
            <end>1407</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Fine-Grained Error Analysis and Fair Evaluation of Labeled Spans
%A Ortmann, Katrin
%S Proceedings of the Thirteenth Language Resources and Evaluation Conference
%D 2022
%8 June
%I European Language Resources Association
%C Marseille, France
%F ortmann-2022-fine
%X The traditional evaluation of labeled spans with precision, recall, and F1-score has undesirable effects due to double penalties. Annotations with incorrect label or boundaries count as two errors instead of one, despite being closer to the target annotation than false positives or false negatives. In this paper, new error types are introduced, which more accurately reflect true annotation quality and ensure that every annotation counts only once. An algorithm for error identification in flat and multi-level annotations is presented and complemented with a proposal on how to calculate meaningful precision, recall, and F1-scores based on the more fine-grained error types. The exemplary application to three different annotation tasks (NER, chunking, parsing) shows that the suggested procedure not only prevents double penalties but also allows for a more detailed error analysis, thereby providing more insight into the actual weaknesses of a system.
%U https://aclanthology.org/2022.lrec-1.150
%P 1400-1407

Markdown (Informal)

[Fine-Grained Error Analysis and Fair Evaluation of Labeled Spans](https://aclanthology.org/2022.lrec-1.150) (Ortmann, LREC 2022)

• Fine-Grained Error Analysis and Fair Evaluation of Labeled Spans (Ortmann, LREC 2022)

ACL

• Katrin Ortmann. 2022. Fine-Grained Error Analysis and Fair Evaluation of Labeled Spans. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 1400–1407, Marseille, France. European Language Resources Association.