LLM-as-judge frameworks are widely used for subjective NLP evaluation, yet their reliability on social-cognition tasks remains unclear. We investigate whether Theory-of-Mind (ToM) scaffolding can help LLM judges evaluate persuasion on Reddit ChangeMyView (CMV). We test four judges (GPT-5, GPT-4.1, Claude Sonnet 4.6, Qwen3-30B-A3B-Instruct-2507) under three conditions that share the same ToM annotations but vary how they are presented: No-ToM, Full-ToM (flat), and our proposed PAToM (Persuasion-Aware ToM), which structures reasoning across five sequential belief-stack turns. Our results show that better reasoning does not lead to better verdicts. Although PAToM improves reasoning coherence by 1.6x, semantic coverage by 1.1-1.8x, and our proposed process metric PRISM by 2-3x, all four judges still struggle to reliably distinguish persuasive comments from high-quality non-persuasive ones. Process-level analysis shows that reasoning quality and verdict accuracy are empirically decoupled under this scaffolding, an open challenge for prompt-based approaches to persuasion judging. We release prompts, ToM annotations, the PRISM metric suite, and a 1000-item benchmark.

This paper outlines our approach to SemEval 2024 Task 9, BRAINTEASER: A Novel Task Defying Common Sense. The task aims to evaluate the ability of language models to think creatively. The dataset comprises multi-choice questions that challenge models to think ‘outside of the box’. We fine-tune 2 models, BERT and RoBERTa Large. Next, we employ a Chain of Thought (CoT) zero-shot prompting approach with 6 large language models, such as GPT-3.5, Mixtral, and Llama2. Finally, we utilize ReConcile, a technique that employs a ‘round table conference’ approach with multiple agents for zero-shot learning, to generate consensus answers among 3 selected language models. Our best method achieves an overall accuracy of 85 percent on the sentence puzzles subtask.