<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="ghiasvand-etal-2026-realm">
    <titleInfo>
        <title>REALM: Reliable Expertise-Aware Language Model Fine-Tuning from Noisy Annotations</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Sajjad</namePart>
        <namePart type="family">Ghiasvand</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Mark</namePart>
        <namePart type="family">Beliaev</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Mahnoosh</namePart>
        <namePart type="family">Alizadeh</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Ramtin</namePart>
        <namePart type="family">Pedarsani</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-10</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: EMNLP 2026</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Hung-Yi</namePart>
            <namePart type="family">Lee</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Sunipa</namePart>
            <namePart type="family">Dev</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jessy</namePart>
            <namePart type="family">Li</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Desmond</namePart>
            <namePart type="family">Elliott</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Budapest, Hungary</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-462-0</identifier>
    </relatedItem>
    <abstract>Supervised fine-tuning of large language models relies on human-annotated data, yet annotation pipelines routinely involve multiple crowdworkers of heterogeneous expertise. Standard practice aggregates labels via majority vote or simple averaging, discarding annotator identity and causing the model to absorb the errors of unreliable annotators into its parameters. We propose REALM, which jointly learns the model parameters and a scalar expertise value for each annotator, entirely unsupervised and requiring nothing beyond annotator identity. The key idea is to model each observed label as a mixture between the model’s prediction and a uniform random guess, weighted by the annotator’s learned expertise. REALM applies to any task with a fixed label set, and extends to multiple tasks via a learned expertise matrix. On four text-classification datasets with real crowdsourced annotations, REALM is the best method in all 12 configurations of the three heterogeneous-annotator datasets, improving on the strongest applicable baseline, including majority vote and Dawid–Skene aggregation, by +2.9 points on average. On five question answering benchmarks with simulated noisy labels, it outperforms naive noisy fine-tuning in 152 of 162 configurations, by +5.0 points on average, with gains that grow with model capacity. The learned expertise additionally recovers annotator reliability without ever observing it. Our code is available at https://github.com/sajjad-ucsb/REALM.</abstract>
    <identifier type="citekey">ghiasvand-etal-2026-realm</identifier>
    <identifier type="doi">10.18653/2026.findings-emnlp.1621</identifier>
    <location>
        <url>https://aclanthology.org/2026.findings-emnlp.1621/</url>
    </location>
    <part>
        <date>2026-10</date>
        <extent unit="page">
            <start>34492</start>
            <end>34509</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T REALM: Reliable Expertise-Aware Language Model Fine-Tuning from Noisy Annotations
%A Ghiasvand, Sajjad
%A Beliaev, Mark
%A Alizadeh, Mahnoosh
%A Pedarsani, Ramtin
%Y Lee, Hung-Yi
%Y Dev, Sunipa
%Y Li, Jessy
%Y Elliott, Desmond
%S Findings of the Association for Computational Linguistics: EMNLP 2026
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-462-0
%F ghiasvand-etal-2026-realm
%X Supervised fine-tuning of large language models relies on human-annotated data, yet annotation pipelines routinely involve multiple crowdworkers of heterogeneous expertise. Standard practice aggregates labels via majority vote or simple averaging, discarding annotator identity and causing the model to absorb the errors of unreliable annotators into its parameters. We propose REALM, which jointly learns the model parameters and a scalar expertise value for each annotator, entirely unsupervised and requiring nothing beyond annotator identity. The key idea is to model each observed label as a mixture between the model’s prediction and a uniform random guess, weighted by the annotator’s learned expertise. REALM applies to any task with a fixed label set, and extends to multiple tasks via a learned expertise matrix. On four text-classification datasets with real crowdsourced annotations, REALM is the best method in all 12 configurations of the three heterogeneous-annotator datasets, improving on the strongest applicable baseline, including majority vote and Dawid–Skene aggregation, by +2.9 points on average. On five question answering benchmarks with simulated noisy labels, it outperforms naive noisy fine-tuning in 152 of 162 configurations, by +5.0 points on average, with gains that grow with model capacity. The learned expertise additionally recovers annotator reliability without ever observing it. Our code is available at https://github.com/sajjad-ucsb/REALM.
%R 10.18653/2026.findings-emnlp.1621
%U https://aclanthology.org/2026.findings-emnlp.1621/
%U https://doi.org/10.18653/2026.findings-emnlp.1621
%P 34492-34509

Markdown (Informal)

[REALM: Reliable Expertise-Aware Language Model Fine-Tuning from Noisy Annotations](https://aclanthology.org/2026.findings-emnlp.1621/) (Ghiasvand et al., Findings 2026)

• REALM: Reliable Expertise-Aware Language Model Fine-Tuning from Noisy Annotations (Ghiasvand et al., Findings 2026)

ACL

• Sajjad Ghiasvand, Mark Beliaev, Mahnoosh Alizadeh, and Ramtin Pedarsani. 2026. REALM: Reliable Expertise-Aware Language Model Fine-Tuning from Noisy Annotations. In Findings of the Association for Computational Linguistics: EMNLP 2026, pages 34492–34509, Budapest, Hungary. Association for Computational Linguistics.