SubmissionNumber#=%=#37
FinalPaperTitle#=%=#Decomposing LLM-Judge–Human Agreement: How Much Is Inherited Surface Preference?
ShortPaperTitle#=%=#
NumberOfPages#=%=#10
CopyrightSigned#=%=#Gaurav Kumar
JobTitle#==#
Organization#==#
Abstract#==#LLM-as-judge evaluation is typically validated using judge-human agreement with standard benchmarks reporting agreement rates of roughly 80-85 %. We show that this aggregate metric can be misleading. Most of that agreement comes from pairs where both judge and human prefer the longer, better-formatted response, rather than from agreement on response quality. On the remaining pairs, agreement drops to chance levels on MT-Bench, and below chance when surface cues mislead. We show this with two simple logistic-regression baselines on length, structure, and lexical features: one predicts the human's pick, one predicts the judge's. The pattern holds across closed and open judges and both benchmarks, and is strongest on the expert annotations the field treats as the quality gold standard. We suggest reporting the inheritance share alongside aggregate agreement: a simple reporting metric we defined that shows whether a judge agrees with humans for the right reasons.
Author{1}{Firstname}#=%=#Gaurav
Author{1}{Lastname}#=%=#Kumar
Author{1}{Username}#=%=#gaurav.k1
Author{1}{Orcid}#=%=#
Author{1}{Email}#=%=#gkumar@ucsd.edu
Author{1}{Affiliation}#=%=#University of California San Diego

==========
èéáğö