<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="wali-tayyab-2026-poll">
    <titleInfo>
        <title>Before You Poll with LLMs: A Deliberative Diagnostic Framework</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Ahmed</namePart>
        <namePart type="family">Wali</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Hassaan</namePart>
        <namePart type="family">Tayyab</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-10</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Hung-Yi</namePart>
            <namePart type="family">Lee</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Sunipa</namePart>
            <namePart type="family">Dev</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jessy</namePart>
            <namePart type="family">Li</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Desmond</namePart>
            <namePart type="family">Elliott</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Budapest, Hungary</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-459-0</identifier>
    </relatedItem>
    <abstract>Can LLMs reason through new information like humans, or do they merely retrieve cached opinions? This question is critical for the growing practice of silicon sampling, where LLM personas simulate public opinion at scale. Current evaluations test only whether these personas hold the right opinions — a static snapshot. But opinion research increasingly depends on dynamic fidelity: whether personas update beliefs in response to new arguments, as humans do during deliberation. No existing benchmark tests this capacity. We introduce the Deliberative Polling Diagnostic Framework, which fills this gap by comparing human and LLM belief shifts after identical informational interventions. Grounded in the methodology of deliberative polling, the framework surfaces failures invisible to static evaluation: models that produce plausible partisan opinions can still misrepresent how those opinions change. Applying the framework to five frontier models using data from America in One Room (526 personas, 72 questions), we observe that every model fails, and in a unique manner. GPT-5.1 exhibits reversal: its personas become more hostile toward the opposing party after balanced information, while humans become less so. This reversal is selective (80% on outgroup vs. 26% on policy questions) and symmetric across partisan identities. Gemini 2.0 Flash, Claude Sonnet 4.5, and Llama 3.3 70B exhibit overshoot, shifting correctly but at 5-7x human magnitude. DeepSeek V3 exhibits rigidity with near-zero change. Targeted ablations reveal that policy content triggers these failures and that they are identity-specific: GPT-5.1 reverses on outgroup questions but overshoots on ingroup; Gemini shows the inverse. We term this behavioral signature self-sycophancy: conformity to the model’s internal stereotype of the persona rather than reasoning from the information provided. Our framework offers practitioners a concrete protocol: run the deliberative diagnostic before trusting LLM personas to accurately mimic revised beliefs.</abstract>
    <identifier type="citekey">wali-tayyab-2026-poll</identifier>
    <location>
        <url>https://aclanthology.org/2026.emnlp-main.2291/</url>
    </location>
    <part>
        <date>2026-10</date>
        <extent unit="page">
            <start>52129</start>
            <end>52145</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Before You Poll with LLMs: A Deliberative Diagnostic Framework
%A Wali, Ahmed
%A Tayyab, Hassaan
%Y Lee, Hung-Yi
%Y Dev, Sunipa
%Y Li, Jessy
%Y Elliott, Desmond
%S Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-459-0
%F wali-tayyab-2026-poll
%X Can LLMs reason through new information like humans, or do they merely retrieve cached opinions? This question is critical for the growing practice of silicon sampling, where LLM personas simulate public opinion at scale. Current evaluations test only whether these personas hold the right opinions — a static snapshot. But opinion research increasingly depends on dynamic fidelity: whether personas update beliefs in response to new arguments, as humans do during deliberation. No existing benchmark tests this capacity. We introduce the Deliberative Polling Diagnostic Framework, which fills this gap by comparing human and LLM belief shifts after identical informational interventions. Grounded in the methodology of deliberative polling, the framework surfaces failures invisible to static evaluation: models that produce plausible partisan opinions can still misrepresent how those opinions change. Applying the framework to five frontier models using data from America in One Room (526 personas, 72 questions), we observe that every model fails, and in a unique manner. GPT-5.1 exhibits reversal: its personas become more hostile toward the opposing party after balanced information, while humans become less so. This reversal is selective (80% on outgroup vs. 26% on policy questions) and symmetric across partisan identities. Gemini 2.0 Flash, Claude Sonnet 4.5, and Llama 3.3 70B exhibit overshoot, shifting correctly but at 5-7x human magnitude. DeepSeek V3 exhibits rigidity with near-zero change. Targeted ablations reveal that policy content triggers these failures and that they are identity-specific: GPT-5.1 reverses on outgroup questions but overshoots on ingroup; Gemini shows the inverse. We term this behavioral signature self-sycophancy: conformity to the model’s internal stereotype of the persona rather than reasoning from the information provided. Our framework offers practitioners a concrete protocol: run the deliberative diagnostic before trusting LLM personas to accurately mimic revised beliefs.
%U https://aclanthology.org/2026.emnlp-main.2291/
%P 52129-52145

Markdown (Informal)

[Before You Poll with LLMs: A Deliberative Diagnostic Framework](https://aclanthology.org/2026.emnlp-main.2291/) (Wali & Tayyab, EMNLP 2026)

• Before You Poll with LLMs: A Deliberative Diagnostic Framework (Wali & Tayyab, EMNLP 2026)

ACL

• Ahmed Wali and Hassaan Tayyab. 2026. Before You Poll with LLMs: A Deliberative Diagnostic Framework. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 52129–52145, Budapest, Hungary. Association for Computational Linguistics.