<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="panahi-etal-2026-llms">
    <titleInfo>
        <title>When LLMs Annotate: Reliability Challenges in Low-Resource NLI</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Solmaz</namePart>
        <namePart type="family">Panahi</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">John</namePart>
        <namePart type="family">Kelleher</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Vasudevan</namePart>
        <namePart type="family">Nedumpozhimana</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-03</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM 2026)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Hansi</namePart>
            <namePart type="family">Hettiarachchi</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Tharindu</namePart>
            <namePart type="family">Ranasinghe</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Alistair</namePart>
            <namePart type="family">Plum</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Paul</namePart>
            <namePart type="family">Rayson</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Ruslan</namePart>
            <namePart type="family">Mitkov</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Mohamed</namePart>
            <namePart type="family">Gaber</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Damith</namePart>
            <namePart type="family">Premasiri</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Fiona</namePart>
            <namePart type="given">Anting</namePart>
            <namePart type="family">Tan</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Lasitha</namePart>
            <namePart type="family">Uyangodage</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Rabat, Morocco</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-377-7</identifier>
    </relatedItem>
    <abstract>This paper systematically evaluates LLM reliability on the complex semantic task of Natural Language Inference (NLI) in Farsi, assessing six prominent models across eight prompt variations through a multi-dimensional framework that measures accuracy, prompt sensitivity, and intra-class consistency. Our results demonstrate that prompt design—particularly the order of premise and hypothesis—significantly impacts prediction stability. Proprietary models (Claude-Opus-4, GPT-4o) exhibit superior stability and accuracy compared to open-weight alternatives. Across all models, the ‘Neutral’ class emerges as the most challenging and least stable category. Crucially, we redefine model instability as a diagnostic tool for benchmark quality, demonstrating that observed disagreement often reflects valid challenges to ambiguous or erroneous gold-standard labels.</abstract>
    <identifier type="citekey">panahi-etal-2026-llms</identifier>
    <location>
        <url>https://aclanthology.org/2026.loreslm-1.17/</url>
    </location>
    <part>
        <date>2026-03</date>
        <extent unit="page">
            <start>178</start>
            <end>188</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T When LLMs Annotate: Reliability Challenges in Low-Resource NLI
%A Panahi, Solmaz
%A Kelleher, John
%A Nedumpozhimana, Vasudevan
%Y Hettiarachchi, Hansi
%Y Ranasinghe, Tharindu
%Y Plum, Alistair
%Y Rayson, Paul
%Y Mitkov, Ruslan
%Y Gaber, Mohamed
%Y Premasiri, Damith
%Y Tan, Fiona Anting
%Y Uyangodage, Lasitha
%S Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM 2026)
%D 2026
%8 March
%I Association for Computational Linguistics
%C Rabat, Morocco
%@ 979-8-89176-377-7
%F panahi-etal-2026-llms
%X This paper systematically evaluates LLM reliability on the complex semantic task of Natural Language Inference (NLI) in Farsi, assessing six prominent models across eight prompt variations through a multi-dimensional framework that measures accuracy, prompt sensitivity, and intra-class consistency. Our results demonstrate that prompt design—particularly the order of premise and hypothesis—significantly impacts prediction stability. Proprietary models (Claude-Opus-4, GPT-4o) exhibit superior stability and accuracy compared to open-weight alternatives. Across all models, the ‘Neutral’ class emerges as the most challenging and least stable category. Crucially, we redefine model instability as a diagnostic tool for benchmark quality, demonstrating that observed disagreement often reflects valid challenges to ambiguous or erroneous gold-standard labels.
%U https://aclanthology.org/2026.loreslm-1.17/
%P 178-188

Markdown (Informal)

[When LLMs Annotate: Reliability Challenges in Low-Resource NLI](https://aclanthology.org/2026.loreslm-1.17/) (Panahi et al., LoResLM 2026)

• When LLMs Annotate: Reliability Challenges in Low-Resource NLI (Panahi et al., LoResLM 2026)

ACL

• Solmaz Panahi, John Kelleher, and Vasudevan Nedumpozhimana. 2026. When LLMs Annotate: Reliability Challenges in Low-Resource NLI. In Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM 2026), pages 178–188, Rabat, Morocco. Association for Computational Linguistics.