<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="keramati-etal-2026-early">
    <titleInfo>
        <title>Early-Token Confidence Predicts Reasoning Quality in Multi-Agent LLM Debate</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Ali</namePart>
        <namePart type="family">Keramati</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Justin</namePart>
        <namePart type="family">Cheok</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Jacob</namePart>
        <namePart type="family">Horne</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Mark</namePart>
        <namePart type="family">Warschauer</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-07</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Simon</namePart>
            <namePart type="family">Mille</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Sebastian</namePart>
            <namePart type="family">Gehrmann</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Patrícia</namePart>
            <namePart type="family">Schmidtová</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Ondřej</namePart>
            <namePart type="family">Dušek</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Marzieh</namePart>
            <namePart type="family">Fadaee</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Kyle</namePart>
            <namePart type="family">Lo</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Enrico</namePart>
            <namePart type="family">Santus</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Gabriel</namePart>
            <namePart type="family">Stanovsky</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">San Diego, California, USA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-423-1</identifier>
    </relatedItem>
    <abstract>Evaluating reasoning quality in multi-agent LLM systems is challenging, especially for open-ended tasks without reference answers. We investigate whether intrinsic confidence signals, token-level log-probabilities from decoding, can predict reasoning quality as assessed by LLM-as-judge evaluation. Using a debate-based essay scoring framework, we compare confidence proxies against rubric-based judge scores across two ASAP essay sets. We find that early-token confidence, particularly within the first few generated tokens, is consistently the strongest predictor of reasoning quality, outperforming full-sequence statistics. Analysis of log-probability trajectories shows that the opening phase of generation is the most heterogeneous and therefore most informative. We also observe a systematic asymmetry between agent roles, with stronger alignment between confidence and quality for supportive reasoning than for adversarial critique. These results suggest that early decoding dynamics provide a lightweight and effective signal for estimating reasoning reliability in multi-agent LLM systems.</abstract>
    <identifier type="citekey">keramati-etal-2026-early</identifier>
    <location>
        <url>https://aclanthology.org/2026.gem-main.60/</url>
    </location>
    <part>
        <date>2026-07</date>
        <extent unit="page">
            <start>653</start>
            <end>667</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Early-Token Confidence Predicts Reasoning Quality in Multi-Agent LLM Debate
%A Keramati, Ali
%A Cheok, Justin
%A Horne, Jacob
%A Warschauer, Mark
%Y Mille, Simon
%Y Gehrmann, Sebastian
%Y Schmidtová, Patrícia
%Y Dušek, Ondřej
%Y Fadaee, Marzieh
%Y Lo, Kyle
%Y Santus, Enrico
%Y Stanovsky, Gabriel
%S Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM)
%D 2026
%8 July
%I Association for Computational Linguistics
%C San Diego, California, USA
%@ 979-8-89176-423-1
%F keramati-etal-2026-early
%X Evaluating reasoning quality in multi-agent LLM systems is challenging, especially for open-ended tasks without reference answers. We investigate whether intrinsic confidence signals, token-level log-probabilities from decoding, can predict reasoning quality as assessed by LLM-as-judge evaluation. Using a debate-based essay scoring framework, we compare confidence proxies against rubric-based judge scores across two ASAP essay sets. We find that early-token confidence, particularly within the first few generated tokens, is consistently the strongest predictor of reasoning quality, outperforming full-sequence statistics. Analysis of log-probability trajectories shows that the opening phase of generation is the most heterogeneous and therefore most informative. We also observe a systematic asymmetry between agent roles, with stronger alignment between confidence and quality for supportive reasoning than for adversarial critique. These results suggest that early decoding dynamics provide a lightweight and effective signal for estimating reasoning reliability in multi-agent LLM systems.
%U https://aclanthology.org/2026.gem-main.60/
%P 653-667

Markdown (Informal)

[Early-Token Confidence Predicts Reasoning Quality in Multi-Agent LLM Debate](https://aclanthology.org/2026.gem-main.60/) (Keramati et al., GEM 2026)

• Early-Token Confidence Predicts Reasoning Quality in Multi-Agent LLM Debate (Keramati et al., GEM 2026)

ACL

• Ali Keramati, Justin Cheok, Jacob Horne, and Mark Warschauer. 2026. Early-Token Confidence Predicts Reasoning Quality in Multi-Agent LLM Debate. In Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM), pages 653–667, San Diego, California, USA. Association for Computational Linguistics.