<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="choppa-etal-2025-predicting">
    <titleInfo>
        <title>Predicting Median, Disagreement and Noise Label in Ordinal Word-in-Context Data</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Tejaswi</namePart>
        <namePart type="family">Choppa</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Michael</namePart>
        <namePart type="family">Roth</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Dominik</namePart>
        <namePart type="family">Schlechtweg</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2025-01</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of Context and Meaning: Navigating Disagreements in NLP Annotation</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Michael</namePart>
            <namePart type="family">Roth</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Dominik</namePart>
            <namePart type="family">Schlechtweg</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>International Committee on Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Abu Dhabi, UAE</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>TThe quality of annotated data is crucial for Machine Learning models, particularly in word sense annotation in context (Word-in-Context, WiC). WiC datasets often show significant annotator disagreement, and information is lost when creating gold labels through majority or median aggregation. Recent work has addressed this by incorporating disagreement data through new label aggregation methods. Modeling disagreement is important since real-world scenarios often lack clean data and require predictions on inherently difficult samples. Disagreement prediction can help detect complex cases or to reflect inherent data ambiguity. We aim to model different aspects of ordinal Word-in-Context annotations necessary to build a more human-like model: (i) the aggregated label, which has traditionally been the modeling aim, (ii) the disagreement between annotators, and (iii) the aggregated noise label which annotators can choose to exclude data points from annotation. We find that disagreement and noise are impacted by various properties of data like ambiguity, which in turn points to data uncertainty.</abstract>
    <identifier type="citekey">choppa-etal-2025-predicting</identifier>
    <location>
        <url>https://aclanthology.org/2025.comedi-1.6/</url>
    </location>
    <part>
        <date>2025-01</date>
        <extent unit="page">
            <start>65</start>
            <end>77</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Predicting Median, Disagreement and Noise Label in Ordinal Word-in-Context Data
%A Choppa, Tejaswi
%A Roth, Michael
%A Schlechtweg, Dominik
%Y Roth, Michael
%Y Schlechtweg, Dominik
%S Proceedings of Context and Meaning: Navigating Disagreements in NLP Annotation
%D 2025
%8 January
%I International Committee on Computational Linguistics
%C Abu Dhabi, UAE
%F choppa-etal-2025-predicting
%X TThe quality of annotated data is crucial for Machine Learning models, particularly in word sense annotation in context (Word-in-Context, WiC). WiC datasets often show significant annotator disagreement, and information is lost when creating gold labels through majority or median aggregation. Recent work has addressed this by incorporating disagreement data through new label aggregation methods. Modeling disagreement is important since real-world scenarios often lack clean data and require predictions on inherently difficult samples. Disagreement prediction can help detect complex cases or to reflect inherent data ambiguity. We aim to model different aspects of ordinal Word-in-Context annotations necessary to build a more human-like model: (i) the aggregated label, which has traditionally been the modeling aim, (ii) the disagreement between annotators, and (iii) the aggregated noise label which annotators can choose to exclude data points from annotation. We find that disagreement and noise are impacted by various properties of data like ambiguity, which in turn points to data uncertainty.
%U https://aclanthology.org/2025.comedi-1.6/
%P 65-77

Markdown (Informal)

[Predicting Median, Disagreement and Noise Label in Ordinal Word-in-Context Data](https://aclanthology.org/2025.comedi-1.6/) (Choppa et al., CoMeDi 2025)

• Predicting Median, Disagreement and Noise Label in Ordinal Word-in-Context Data (Choppa et al., CoMeDi 2025)

ACL

• Tejaswi Choppa, Michael Roth, and Dominik Schlechtweg. 2025. Predicting Median, Disagreement and Noise Label in Ordinal Word-in-Context Data. In Proceedings of Context and Meaning: Navigating Disagreements in NLP Annotation, pages 65–77, Abu Dhabi, UAE. International Committee on Computational Linguistics.