<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="madhusudhan-etal-2025-llms">
    <titleInfo>
        <title>Do LLMs Know When to NOT Answer? Investigating Abstention Abilities of Large Language Models</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Nishanth</namePart>
        <namePart type="family">Madhusudhan</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Sathwik</namePart>
        <namePart type="given">Tejaswi</namePart>
        <namePart type="family">Madhusudhan</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Vikas</namePart>
        <namePart type="family">Yadav</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Masoud</namePart>
        <namePart type="family">Hashemi</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2025-01</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 31st International Conference on Computational Linguistics</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Owen</namePart>
            <namePart type="family">Rambow</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Leo</namePart>
            <namePart type="family">Wanner</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Marianna</namePart>
            <namePart type="family">Apidianaki</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Hend</namePart>
            <namePart type="family">Al-Khalifa</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Barbara</namePart>
            <namePart type="given">Di</namePart>
            <namePart type="family">Eugenio</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Steven</namePart>
            <namePart type="family">Schockaert</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Abu Dhabi, UAE</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Abstention Ability (AA) is a critical aspect of Large Language Model (LLM) reliability, referring to an LLM‘s capability to withhold responses when uncertain or lacking a definitive answer, without compromising performance. Although previous studies have attempted to improve AA, they lack a standardized evaluation method and remain unsuitable for black-box models where token prediction probabilities are inaccessible. This makes comparative analysis challenging, especially for state-of-the-art closed-source commercial LLMs. This paper bridges this gap by introducing a black-box evaluation approach and a new dataset, Abstain-QA, crafted to rigorously assess AA across varied question types (answerable and unanswerable), domains (well-represented and under-represented), and task types (fact-centric and reasoning). We also propose a new confusion matrix, the ”Answerable-Unanswerable Confusion Matrix” (AUCM) which serves as the basis for evaluating AA, by offering a structured and precise approach for assessment. Finally, we explore the impact of three prompting strategies — Strict Prompting, Verbal Confidence Thresholding, and Chain-of-Thought (CoT) — on improving AA. Our results indicate that even powerful models like GPT-4, Mixtral 8x22b encounter difficulties with abstention; however, strategic approaches such as Strict prompting and CoT can enhance this capability.</abstract>
    <identifier type="citekey">madhusudhan-etal-2025-llms</identifier>
    <location>
        <url>https://aclanthology.org/2025.coling-main.627/</url>
    </location>
    <part>
        <date>2025-01</date>
        <extent unit="page">
            <start>9329</start>
            <end>9345</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Do LLMs Know When to NOT Answer? Investigating Abstention Abilities of Large Language Models
%A Madhusudhan, Nishanth
%A Madhusudhan, Sathwik Tejaswi
%A Yadav, Vikas
%A Hashemi, Masoud
%Y Rambow, Owen
%Y Wanner, Leo
%Y Apidianaki, Marianna
%Y Al-Khalifa, Hend
%Y Eugenio, Barbara Di
%Y Schockaert, Steven
%S Proceedings of the 31st International Conference on Computational Linguistics
%D 2025
%8 January
%I Association for Computational Linguistics
%C Abu Dhabi, UAE
%F madhusudhan-etal-2025-llms
%X Abstention Ability (AA) is a critical aspect of Large Language Model (LLM) reliability, referring to an LLM‘s capability to withhold responses when uncertain or lacking a definitive answer, without compromising performance. Although previous studies have attempted to improve AA, they lack a standardized evaluation method and remain unsuitable for black-box models where token prediction probabilities are inaccessible. This makes comparative analysis challenging, especially for state-of-the-art closed-source commercial LLMs. This paper bridges this gap by introducing a black-box evaluation approach and a new dataset, Abstain-QA, crafted to rigorously assess AA across varied question types (answerable and unanswerable), domains (well-represented and under-represented), and task types (fact-centric and reasoning). We also propose a new confusion matrix, the ”Answerable-Unanswerable Confusion Matrix” (AUCM) which serves as the basis for evaluating AA, by offering a structured and precise approach for assessment. Finally, we explore the impact of three prompting strategies — Strict Prompting, Verbal Confidence Thresholding, and Chain-of-Thought (CoT) — on improving AA. Our results indicate that even powerful models like GPT-4, Mixtral 8x22b encounter difficulties with abstention; however, strategic approaches such as Strict prompting and CoT can enhance this capability.
%U https://aclanthology.org/2025.coling-main.627/
%P 9329-9345

Markdown (Informal)

[Do LLMs Know When to NOT Answer? Investigating Abstention Abilities of Large Language Models](https://aclanthology.org/2025.coling-main.627/) (Madhusudhan et al., COLING 2025)

• Do LLMs Know When to NOT Answer? Investigating Abstention Abilities of Large Language Models (Madhusudhan et al., COLING 2025)

ACL

• Nishanth Madhusudhan, Sathwik Tejaswi Madhusudhan, Vikas Yadav, and Masoud Hashemi. 2025. Do LLMs Know When to NOT Answer? Investigating Abstention Abilities of Large Language Models. In Proceedings of the 31st International Conference on Computational Linguistics, pages 9329–9345, Abu Dhabi, UAE. Association for Computational Linguistics.