Afra Mashhadi
Author directory2026
Reasoning Well, Judging Poorly: Theory of Mind Scaffolding Exposes the Limits of LLM Persuasion Judges
Baktash Ansari | Mouly Dewan | Afra Mashhadi
Findings of the Association for Computational Linguistics: EMNLP 2026
Baktash Ansari | Mouly Dewan | Afra Mashhadi
Findings of the Association for Computational Linguistics: EMNLP 2026
LLM-as-judge frameworks are widely used for subjective NLP evaluation, yet their reliability on social-cognition tasks remains unclear. We investigate whether Theory-of-Mind (ToM) scaffolding can help LLM judges evaluate persuasion on Reddit ChangeMyView (CMV). We test four judges (GPT-5, GPT-4.1, Claude Sonnet 4.6, Qwen3-30B-A3B-Instruct-2507) under three conditions that share the same ToM annotations but vary how they are presented: No-ToM, Full-ToM (flat), and our proposed PAToM (Persuasion-Aware ToM), which structures reasoning across five sequential belief-stack turns. Our results show that better reasoning does not lead to better verdicts. Although PAToM improves reasoning coherence by 1.6x, semantic coverage by 1.1-1.8x, and our proposed process metric PRISM by 2-3x, all four judges still struggle to reliably distinguish persuasive comments from high-quality non-persuasive ones. Process-level analysis shows that reasoning quality and verdict accuracy are empirically decoupled under this scaffolding, an open challenge for prompt-based approaches to persuasion judging. We release prompts, ToM annotations, the PRISM metric suite, and a 1000-item benchmark.