Zeyu He
2026
PERSONAJUDGE: Simulating Individual Human Preference Judgments with Evaluator-Specific Demonstration Data
Zeyu He | Xuan Qi | Subramanian Chidambaram | Zhichao Xu | Vinayak Arannil | Lydia Chilton | Alex C. Williams
Proceedings of the 27th Annual Meeting of the Special Interest Group on Discourse and Dialogue
Zeyu He | Xuan Qi | Subramanian Chidambaram | Zhichao Xu | Vinayak Arannil | Lydia Chilton | Alex C. Williams
Proceedings of the 27th Annual Meeting of the Special Interest Group on Discourse and Dialogue
Large language models increasingly serve as judges in AI evaluation, but current approaches rely on consensus preferences that ignore individual evaluator variation. We propose a novel simulation approach that combines categorical judgments with evaluator-specific auxiliary data—retrospective reasoning traces and interface telemetry—to enable LLM-based simulation of individual evaluators via in-context learning. We conduct a systematic empirical study of this approach using multi-facet data from 32 trained annotators across 4,200 preference judgments in a 4 × 4 × 4 factorial design. Our key findings: (1) The simulation approach achieves up to 9.9 percentage point improvements over the Base Judge; (2) Reasoning traces provide the largest gains with higher collection efforts, while interface telemetry often hurts rather than helps performance despite being cheaper to collect. (3) Simulation difficulty is systematic, predicted by an evaluator’s neutral usage (most clearly on Helpfulness) and divergence from consensus; the neutral-usage tendency—rather than simulatability itself—is the cross-task-stable property (r = 0.728). These results establish both the potential and limits of evaluator-specific auxiliary data for personalized evaluation, offering methodological insights for scaling individual aware AI assessment.