@inproceedings{ravideshik-kejriwal-2026-reliability,
title = "Reliability of {LLM} Judges for Evaluating Entity Alignment",
author = "Ravideshik, Vaibhava Lakshmi and
Kejriwal, Mayank",
editor = "Lee, Hung-Yi and
Dev, Sunipa and
Li, Jessy and
Elliott, Desmond",
booktitle = "Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing",
month = oct,
year = "2026",
address = "Budapest, Hungary",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2026.emnlp-main.1717/",
pages = "39175--39200",
ISBN = "979-8-89176-459-0",
abstract = "Entity Alignment (EA) identifies equivalent entities across knowledge graphs and is critical for knowledge base integration and ontology merging. Evaluating EA systems at scale requires expensive expert annotation, making systematic assessment across diverse domains practically infeasible. LLM-as-judge evaluation offers a potentially scalable alternative, yet its reliability for structured prediction tasks like EA remains unstudied. We present the first systematic benchmarking study across three frontier models, three datasets, and four EA systems, using perturbation bias diagnostics, meta-evaluation across all dataset{--}judge{--}prompt combinations, and counterfactual label-flip tests. We identify anchor bias, a failure mode in which judges invert discrimination when the system{'}s decision label is visible. Label exposure causally collapses judge discrimination (J-ROC-AUC 0.12{--}0.87), while a label-free protocol recovers near-ceiling capability on distinctive-name datasets (0.93{--}1.00) and significant recovery on biomedical pairs (0.93{--}0.95). Counterfactual experiments confirm causality (FSR 53{--}99{\%}) and reveal a frontier model paradox: stronger judges exhibit greater label sensitivity, not less. A blinded two-annotator human evaluation (102 pairs, Cohen{'}s {\ensuremath{\kappa}} = 0.902) confirms this mechanism directly. We release the first biomedical EA benchmark (MeSH{--}SNOMED CT, 15K pairs) and a reproducible auditing framework for LLM judge reliability in EA. Code and data are available at https://github.com/vaibhavalakshmiravideshik/llm-as-a-judge-entity-alignment."
}<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="ravideshik-kejriwal-2026-reliability">
<titleInfo>
<title>Reliability of LLM Judges for Evaluating Entity Alignment</title>
</titleInfo>
<name type="personal">
<namePart type="given">Vaibhava</namePart>
<namePart type="given">Lakshmi</namePart>
<namePart type="family">Ravideshik</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Mayank</namePart>
<namePart type="family">Kejriwal</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<originInfo>
<dateIssued>2026-10</dateIssued>
</originInfo>
<typeOfResource>text</typeOfResource>
<relatedItem type="host">
<titleInfo>
<title>Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing</title>
</titleInfo>
<name type="personal">
<namePart type="given">Hung-Yi</namePart>
<namePart type="family">Lee</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Sunipa</namePart>
<namePart type="family">Dev</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Jessy</namePart>
<namePart type="family">Li</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Desmond</namePart>
<namePart type="family">Elliott</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<originInfo>
<publisher>Association for Computational Linguistics</publisher>
<place>
<placeTerm type="text">Budapest, Hungary</placeTerm>
</place>
</originInfo>
<genre authority="marcgt">conference publication</genre>
<identifier type="isbn">979-8-89176-459-0</identifier>
</relatedItem>
<abstract>Entity Alignment (EA) identifies equivalent entities across knowledge graphs and is critical for knowledge base integration and ontology merging. Evaluating EA systems at scale requires expensive expert annotation, making systematic assessment across diverse domains practically infeasible. LLM-as-judge evaluation offers a potentially scalable alternative, yet its reliability for structured prediction tasks like EA remains unstudied. We present the first systematic benchmarking study across three frontier models, three datasets, and four EA systems, using perturbation bias diagnostics, meta-evaluation across all dataset–judge–prompt combinations, and counterfactual label-flip tests. We identify anchor bias, a failure mode in which judges invert discrimination when the system’s decision label is visible. Label exposure causally collapses judge discrimination (J-ROC-AUC 0.12–0.87), while a label-free protocol recovers near-ceiling capability on distinctive-name datasets (0.93–1.00) and significant recovery on biomedical pairs (0.93–0.95). Counterfactual experiments confirm causality (FSR 53–99%) and reveal a frontier model paradox: stronger judges exhibit greater label sensitivity, not less. A blinded two-annotator human evaluation (102 pairs, Cohen’s \ensuremathąppa = 0.902) confirms this mechanism directly. We release the first biomedical EA benchmark (MeSH–SNOMED CT, 15K pairs) and a reproducible auditing framework for LLM judge reliability in EA. Code and data are available at https://github.com/vaibhavalakshmiravideshik/llm-as-a-judge-entity-alignment.</abstract>
<identifier type="citekey">ravideshik-kejriwal-2026-reliability</identifier>
<location>
<url>https://aclanthology.org/2026.emnlp-main.1717/</url>
</location>
<part>
<date>2026-10</date>
<extent unit="page">
<start>39175</start>
<end>39200</end>
</extent>
</part>
</mods>
</modsCollection>
%0 Conference Proceedings
%T Reliability of LLM Judges for Evaluating Entity Alignment
%A Ravideshik, Vaibhava Lakshmi
%A Kejriwal, Mayank
%Y Lee, Hung-Yi
%Y Dev, Sunipa
%Y Li, Jessy
%Y Elliott, Desmond
%S Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-459-0
%F ravideshik-kejriwal-2026-reliability
%X Entity Alignment (EA) identifies equivalent entities across knowledge graphs and is critical for knowledge base integration and ontology merging. Evaluating EA systems at scale requires expensive expert annotation, making systematic assessment across diverse domains practically infeasible. LLM-as-judge evaluation offers a potentially scalable alternative, yet its reliability for structured prediction tasks like EA remains unstudied. We present the first systematic benchmarking study across three frontier models, three datasets, and four EA systems, using perturbation bias diagnostics, meta-evaluation across all dataset–judge–prompt combinations, and counterfactual label-flip tests. We identify anchor bias, a failure mode in which judges invert discrimination when the system’s decision label is visible. Label exposure causally collapses judge discrimination (J-ROC-AUC 0.12–0.87), while a label-free protocol recovers near-ceiling capability on distinctive-name datasets (0.93–1.00) and significant recovery on biomedical pairs (0.93–0.95). Counterfactual experiments confirm causality (FSR 53–99%) and reveal a frontier model paradox: stronger judges exhibit greater label sensitivity, not less. A blinded two-annotator human evaluation (102 pairs, Cohen’s \ensuremathąppa = 0.902) confirms this mechanism directly. We release the first biomedical EA benchmark (MeSH–SNOMED CT, 15K pairs) and a reproducible auditing framework for LLM judge reliability in EA. Code and data are available at https://github.com/vaibhavalakshmiravideshik/llm-as-a-judge-entity-alignment.
%U https://aclanthology.org/2026.emnlp-main.1717/
%P 39175-39200
Markdown (Informal)
[Reliability of LLM Judges for Evaluating Entity Alignment](https://aclanthology.org/2026.emnlp-main.1717/) (Ravideshik & Kejriwal, EMNLP 2026)
ACL
- Vaibhava Lakshmi Ravideshik and Mayank Kejriwal. 2026. Reliability of LLM Judges for Evaluating Entity Alignment. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 39175–39200, Budapest, Hungary. Association for Computational Linguistics.