<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="kurniawan-etal-2026-training">
    <titleInfo>
        <title>Training and Evaluating with Human Label Variation: An Empirical Study</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Kemal</namePart>
        <namePart type="family">Kurniawan</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Meladel</namePart>
        <namePart type="family">Mistica</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Timothy</namePart>
        <namePart type="family">Baldwin</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Jey</namePart>
        <namePart type="given">Han</namePart>
        <namePart type="family">Lau</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-03</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <genre authority="bibutilsgt">journal article</genre>
    <relatedItem type="host">
        <titleInfo>
            <title>Computational Linguistics</title>
        </titleInfo>
        <originInfo>
            <issuance>continuing</issuance>
            <publisher>MIT Press</publisher>
            <place>
                <placeTerm type="text">Cambridge, MA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">periodical</genre>
        <genre authority="bibutilsgt">academic journal</genre>
    </relatedItem>
    <abstract>Human label variation (HLV) challenges the standard assumption that a labeled instance has a single ground truth, instead embracing the natural variation in human annotation to train and evaluate models. While various training methods and metrics for HLV have been proposed, it is still unclear which methods and metrics perform best in what settings. We propose new evaluation metrics for HLV leveraging fuzzy set theory. Because these new proposed metrics are differentiable, we then in turn experiment with using these metrics as training objectives. We conduct an extensive study over 6 HLV datasets testing 14 training methods and 6 evaluation metrics. We find that training on either disaggregated annotations or soft labels performs best across metrics, outperforming training using the proposed training objectives with differentiable metrics. We also show that our proposed soft micro F1 score is one of the best metrics for HLV data.1</abstract>
    <identifier type="citekey">kurniawan-etal-2026-training</identifier>
    <identifier type="doi">10.1162/coli.a.578</identifier>
    <location>
        <url>https://aclanthology.org/2026.cl-1.3/</url>
    </location>
    <part>
        <date>2026-03</date>
        <detail type="volume"><number>52</number></detail>
        <detail type="issue"><number>1</number></detail>
        <extent unit="page">
            <start>85</start>
            <end>111</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Journal Article
%T Training and Evaluating with Human Label Variation: An Empirical Study
%A Kurniawan, Kemal
%A Mistica, Meladel
%A Baldwin, Timothy
%A Lau, Jey Han
%J Computational Linguistics
%D 2026
%8 March
%V 52
%N 1
%I MIT Press
%C Cambridge, MA
%F kurniawan-etal-2026-training
%X Human label variation (HLV) challenges the standard assumption that a labeled instance has a single ground truth, instead embracing the natural variation in human annotation to train and evaluate models. While various training methods and metrics for HLV have been proposed, it is still unclear which methods and metrics perform best in what settings. We propose new evaluation metrics for HLV leveraging fuzzy set theory. Because these new proposed metrics are differentiable, we then in turn experiment with using these metrics as training objectives. We conduct an extensive study over 6 HLV datasets testing 14 training methods and 6 evaluation metrics. We find that training on either disaggregated annotations or soft labels performs best across metrics, outperforming training using the proposed training objectives with differentiable metrics. We also show that our proposed soft micro F1 score is one of the best metrics for HLV data.1
%R 10.1162/coli.a.578
%U https://aclanthology.org/2026.cl-1.3/
%U https://doi.org/10.1162/coli.a.578
%P 85-111

Markdown (Informal)

[Training and Evaluating with Human Label Variation: An Empirical Study](https://aclanthology.org/2026.cl-1.3/) (Kurniawan et al., CL 2026)

• Training and Evaluating with Human Label Variation: An Empirical Study (Kurniawan et al., CL 2026)

ACL

• Kemal Kurniawan, Meladel Mistica, Timothy Baldwin, and Jey Han Lau. 2026. Training and Evaluating with Human Label Variation: An Empirical Study. Computational Linguistics, 52(1):85–111.