<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="kuklin-arefyev-2025-deep">
    <titleInfo>
        <title>Deep-change at CoMeDi: the Cross-Entropy Loss is not All You Need</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Mikhail</namePart>
        <namePart type="family">Kuklin</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Nikolay</namePart>
        <namePart type="family">Arefyev</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2025-01</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of Context and Meaning: Navigating Disagreements in NLP Annotation</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Michael</namePart>
            <namePart type="family">Roth</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Dominik</namePart>
            <namePart type="family">Schlechtweg</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>International Committee on Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Abu Dhabi, UAE</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Manual annotation of edges in Diachronic Word Usage Graphs is a critical step in creation of datasets for Lexical Semantic Change Detection tasks, but a very labour-intensive one. Annotators estimate if two senses of an ambiguous word expressed in two usages of this word are related and how. This is a variation of the Word-in-Context (WiC) task with some peculiarities, including diachronic data, an ordinal scale for annotations consisting of 4 values with pre-defined meanings (e.g. homonymy, polysemy), and special attention to the degree of disagreement between annotators which affects the further processing of the graph. CoMeDi is a shared task aiming at automating this annotation process. Participants are asked to predict the median annotation for a pair of usages in the first subtask, and estimate the disagreement between annotators in the second subtask. Together this gives some idea about the distribution of annotations we can get from humans for a given pair of usages. For the first subtask we tried several ways of adapting a binary WiC model to this 4 class problem. We discovered that further fine-tuning the model as a 4 class classifier on the training data of the shared task works significantly worse than thresholding the original binary model. For the second subtask our best results were achieved by building a model that predicts the whole multinomial distribution of annotations and calculating the disagreement from this distribution. Our solutions for both subtasks have outperformed all other participants of the shared task.</abstract>
    <identifier type="citekey">kuklin-arefyev-2025-deep</identifier>
    <location>
        <url>https://aclanthology.org/2025.comedi-1.5/</url>
    </location>
    <part>
        <date>2025-01</date>
        <extent unit="page">
            <start>48</start>
            <end>64</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Deep-change at CoMeDi: the Cross-Entropy Loss is not All You Need
%A Kuklin, Mikhail
%A Arefyev, Nikolay
%Y Roth, Michael
%Y Schlechtweg, Dominik
%S Proceedings of Context and Meaning: Navigating Disagreements in NLP Annotation
%D 2025
%8 January
%I International Committee on Computational Linguistics
%C Abu Dhabi, UAE
%F kuklin-arefyev-2025-deep
%X Manual annotation of edges in Diachronic Word Usage Graphs is a critical step in creation of datasets for Lexical Semantic Change Detection tasks, but a very labour-intensive one. Annotators estimate if two senses of an ambiguous word expressed in two usages of this word are related and how. This is a variation of the Word-in-Context (WiC) task with some peculiarities, including diachronic data, an ordinal scale for annotations consisting of 4 values with pre-defined meanings (e.g. homonymy, polysemy), and special attention to the degree of disagreement between annotators which affects the further processing of the graph. CoMeDi is a shared task aiming at automating this annotation process. Participants are asked to predict the median annotation for a pair of usages in the first subtask, and estimate the disagreement between annotators in the second subtask. Together this gives some idea about the distribution of annotations we can get from humans for a given pair of usages. For the first subtask we tried several ways of adapting a binary WiC model to this 4 class problem. We discovered that further fine-tuning the model as a 4 class classifier on the training data of the shared task works significantly worse than thresholding the original binary model. For the second subtask our best results were achieved by building a model that predicts the whole multinomial distribution of annotations and calculating the disagreement from this distribution. Our solutions for both subtasks have outperformed all other participants of the shared task.
%U https://aclanthology.org/2025.comedi-1.5/
%P 48-64

Markdown (Informal)

[Deep-change at CoMeDi: the Cross-Entropy Loss is not All You Need](https://aclanthology.org/2025.comedi-1.5/) (Kuklin & Arefyev, CoMeDi 2025)

• Deep-change at CoMeDi: the Cross-Entropy Loss is not All You Need (Kuklin & Arefyev, CoMeDi 2025)

ACL

• Mikhail Kuklin and Nikolay Arefyev. 2025. Deep-change at CoMeDi: the Cross-Entropy Loss is not All You Need. In Proceedings of Context and Meaning: Navigating Disagreements in NLP Annotation, pages 48–64, Abu Dhabi, UAE. International Committee on Computational Linguistics.