Hassaan Tayyab

Author directory

2026

Can LLMs reason through new information like humans, or do they merely retrieve cached opinions? This question is critical for the growing practice of silicon sampling, where LLM personas simulate public opinion at scale. Current evaluations test only whether these personas hold the right opinions — a static snapshot. But opinion research increasingly depends on dynamic fidelity: whether personas update beliefs in response to new arguments, as humans do during deliberation. No existing benchmark tests this capacity. We introduce the Deliberative Polling Diagnostic Framework, which fills this gap by comparing human and LLM belief shifts after identical informational interventions. Grounded in the methodology of deliberative polling, the framework surfaces failures invisible to static evaluation: models that produce plausible partisan opinions can still misrepresent how those opinions change. Applying the framework to five frontier models using data from America in One Room (526 personas, 72 questions), we observe that every model fails, and in a unique manner. GPT-5.1 exhibits reversal: its personas become more hostile toward the opposing party after balanced information, while humans become less so. This reversal is selective (80% on outgroup vs. 26% on policy questions) and symmetric across partisan identities. Gemini 2.0 Flash, Claude Sonnet 4.5, and Llama 3.3 70B exhibit overshoot, shifting correctly but at 5-7x human magnitude. DeepSeek V3 exhibits rigidity with near-zero change. Targeted ablations reveal that policy content triggers these failures and that they are identity-specific: GPT-5.1 reverses on outgroup questions but overshoots on ingroup; Gemini shows the inverse. We term this behavioral signature self-sycophancy: conformity to the model’s internal stereotype of the persona rather than reasoning from the information provided. Our framework offers practitioners a concrete protocol: run the deliberative diagnostic before trusting LLM personas to accurately mimic revised beliefs.