<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="kulmizev-nivre-2023-investigating">
    <titleInfo>
        <title>Investigating UD Treebanks via Dataset Difficulty Measures</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Artur</namePart>
        <namePart type="family">Kulmizev</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Joakim</namePart>
        <namePart type="family">Nivre</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-05</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Andreas</namePart>
            <namePart type="family">Vlachos</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Isabelle</namePart>
            <namePart type="family">Augenstein</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Dubrovnik, Croatia</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Treebanks annotated with Universal Dependencies (UD) are currently available for over 100 languages and are widely utilized by the community. However, their inherent characteristics are hard to measure and are only partially reflected in parser evaluations via accuracy metrics like LAS. In this study, we analyze a large subset of the UD treebanks using three recently proposed accuracy-free dataset analysis methods: dataset cartography, \mathcalV-information, and minimum description length. Each method provides insights about UD treebanks that would remain undetected if only LAS was considered. Specifically, we identify a number of treebanks that, despite yielding high LAS, contain very little information that is usable by a parser to surpass what can be achieved by simple heuristics. Furthermore, we make note of several treebanks that score consistently low across numerous metrics, indicating a high degree of noise or annotation inconsistency present therein.</abstract>
    <identifier type="citekey">kulmizev-nivre-2023-investigating</identifier>
    <identifier type="doi">10.18653/v1/2023.eacl-main.76</identifier>
    <location>
        <url>https://aclanthology.org/2023.eacl-main.76</url>
    </location>
    <part>
        <date>2023-05</date>
        <extent unit="page">
            <start>1076</start>
            <end>1089</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Investigating UD Treebanks via Dataset Difficulty Measures
%A Kulmizev, Artur
%A Nivre, Joakim
%Y Vlachos, Andreas
%Y Augenstein, Isabelle
%S Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics
%D 2023
%8 May
%I Association for Computational Linguistics
%C Dubrovnik, Croatia
%F kulmizev-nivre-2023-investigating
%X Treebanks annotated with Universal Dependencies (UD) are currently available for over 100 languages and are widely utilized by the community. However, their inherent characteristics are hard to measure and are only partially reflected in parser evaluations via accuracy metrics like LAS. In this study, we analyze a large subset of the UD treebanks using three recently proposed accuracy-free dataset analysis methods: dataset cartography, \mathcalV-information, and minimum description length. Each method provides insights about UD treebanks that would remain undetected if only LAS was considered. Specifically, we identify a number of treebanks that, despite yielding high LAS, contain very little information that is usable by a parser to surpass what can be achieved by simple heuristics. Furthermore, we make note of several treebanks that score consistently low across numerous metrics, indicating a high degree of noise or annotation inconsistency present therein.
%R 10.18653/v1/2023.eacl-main.76
%U https://aclanthology.org/2023.eacl-main.76
%U https://doi.org/10.18653/v1/2023.eacl-main.76
%P 1076-1089

Markdown (Informal)

[Investigating UD Treebanks via Dataset Difficulty Measures](https://aclanthology.org/2023.eacl-main.76) (Kulmizev & Nivre, EACL 2023)

• Investigating UD Treebanks via Dataset Difficulty Measures (Kulmizev & Nivre, EACL 2023)

ACL

• Artur Kulmizev and Joakim Nivre. 2023. Investigating UD Treebanks via Dataset Difficulty Measures. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1076–1089, Dubrovnik, Croatia. Association for Computational Linguistics.