Daniel M. Bikel
Author directoryOther people with similar names: Daniel M. Bikel
2026
Auditing LLM Benchmarks with Item Response Theory
Sander Land | Daniel M. Bikel
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Sander Land | Daniel M. Bikel
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
LLM benchmark labels are frozen at release and silently copied into downstream benchmarks, errors and all. Our Item Response Theory indicator surfaces these mislabels at 95% precision in the top 200, across 114 models on seven preference and multiple-choice benchmarks, outperforming a supervised classifier. We trace these errors to mechanical labeling rules, upstream errors inherited unchanged from source datasets, and ambiguous items with no defensible single key. The same fit reveals that reward models specialize in stylistic preference over factual knowledge, and exposes one frontier reward model that agrees with detected mislabels at 78% accuracy versus 38% for its peers, consistent with contamination or benchmark-specific over-optimization.