<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="guo-etal-2025-benchmarking">
    <titleInfo>
        <title>Benchmarking Uncertainty Metrics for LLM Target-Aware Search</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Pei-Fu</namePart>
        <namePart type="family">Guo</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Yun-Da</namePart>
        <namePart type="family">Tsai</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Shou-De</namePart>
        <namePart type="family">Lin</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2025-11</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: EMNLP 2025</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Christos</namePart>
            <namePart type="family">Christodoulopoulos</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Tanmoy</namePart>
            <namePart type="family">Chakraborty</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Carolyn</namePart>
            <namePart type="family">Rose</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Violet</namePart>
            <namePart type="family">Peng</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Suzhou, China</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-335-7</identifier>
    </relatedItem>
    <abstract>LLM search methods, such as Chain of Thought (CoT) and Tree of Thought (ToT), enhance LLM reasoning by exploring multiple reasoning paths. When combined with search algorithms like MCTS and Bandit methods, their effectiveness relies heavily on uncertainty estimation to prioritize paths that align with specific search objectives. However, it remains unclear whether existing LLM uncertainty metrics adequately capture the diverse types of uncertainty required to guide different search objectives. In this work, we introduce a framework for uncertainty benchmarking, identifying four distinct uncertainty types: Answer, Correctness, Aleatoric, and Epistemic Uncertainty. Each type serves different optimization goals in search. Our experiments demonstrate that current metrics often align with only a subset of these uncertainty types, limiting their effectiveness for objective-aligned search in some cases. These findings highlight the need for additional target-aware uncertainty estimators that can adapt to various optimization goals in LLM search.</abstract>
    <identifier type="citekey">guo-etal-2025-benchmarking</identifier>
    <identifier type="doi">10.18653/v1/2025.findings-emnlp.226</identifier>
    <location>
        <url>https://aclanthology.org/2025.findings-emnlp.226/</url>
    </location>
    <part>
        <date>2025-11</date>
        <extent unit="page">
            <start>4230</start>
            <end>4238</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Benchmarking Uncertainty Metrics for LLM Target-Aware Search
%A Guo, Pei-Fu
%A Tsai, Yun-Da
%A Lin, Shou-De
%Y Christodoulopoulos, Christos
%Y Chakraborty, Tanmoy
%Y Rose, Carolyn
%Y Peng, Violet
%S Findings of the Association for Computational Linguistics: EMNLP 2025
%D 2025
%8 November
%I Association for Computational Linguistics
%C Suzhou, China
%@ 979-8-89176-335-7
%F guo-etal-2025-benchmarking
%X LLM search methods, such as Chain of Thought (CoT) and Tree of Thought (ToT), enhance LLM reasoning by exploring multiple reasoning paths. When combined with search algorithms like MCTS and Bandit methods, their effectiveness relies heavily on uncertainty estimation to prioritize paths that align with specific search objectives. However, it remains unclear whether existing LLM uncertainty metrics adequately capture the diverse types of uncertainty required to guide different search objectives. In this work, we introduce a framework for uncertainty benchmarking, identifying four distinct uncertainty types: Answer, Correctness, Aleatoric, and Epistemic Uncertainty. Each type serves different optimization goals in search. Our experiments demonstrate that current metrics often align with only a subset of these uncertainty types, limiting their effectiveness for objective-aligned search in some cases. These findings highlight the need for additional target-aware uncertainty estimators that can adapt to various optimization goals in LLM search.
%R 10.18653/v1/2025.findings-emnlp.226
%U https://aclanthology.org/2025.findings-emnlp.226/
%U https://doi.org/10.18653/v1/2025.findings-emnlp.226
%P 4230-4238

Markdown (Informal)

[Benchmarking Uncertainty Metrics for LLM Target-Aware Search](https://aclanthology.org/2025.findings-emnlp.226/) (Guo et al., Findings 2025)

• Benchmarking Uncertainty Metrics for LLM Target-Aware Search (Guo et al., Findings 2025)

ACL

• Pei-Fu Guo, Yun-Da Tsai, and Shou-De Lin. 2025. Benchmarking Uncertainty Metrics for LLM Target-Aware Search. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 4230–4238, Suzhou, China. Association for Computational Linguistics.