Anchored Bradley-Terry Calibration Using LLM Comparative Judgments

Ummugul Bezirhan, Matthias von Davier


Abstract
AI-based comparative judgment was evaluated as a tool for early item difficulty estimation. Three AI judges compared new TIMSS Grade 4 mathematics items with calibrated anchors, with rankings analyzed using a fixed-anchor Bradley-Terry model. Results show meaningful difficulty signals, supporting scalable supplementary use while highlighting anchor coverage and comparison-network design.
Anthology ID:
2026.aimecon-main.40
Volume:
Proceedings of the Artificial Intelligence in Measurement and Education Conference (AIME-Con): Full Papers
Month:
October
Year:
2026
Address:
Wyndham Grand Pittsburgh Downtown, Pittsburgh, Pennsylvania, United States
Editors:
Joshua Wilson, Christopher Ormerod, Magdalen Beiting-Parrish
Venue:
AIME-Con
SIG:
Publisher:
National Council on Measurement in Education (NCME)
Note:
Pages:
365–371
Language:
URL:
https://aclanthology.org/2026.aimecon-main.40/
DOI:
Bibkey:
Cite (ACL):
Ummugul Bezirhan and Matthias von Davier. 2026. Anchored Bradley-Terry Calibration Using LLM Comparative Judgments. In Proceedings of the Artificial Intelligence in Measurement and Education Conference (AIME-Con): Full Papers, pages 365–371, Wyndham Grand Pittsburgh Downtown, Pittsburgh, Pennsylvania, United States. National Council on Measurement in Education (NCME).
Cite (Informal):
Anchored Bradley-Terry Calibration Using LLM Comparative Judgments (Bezirhan & von Davier, AIME-Con 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.aimecon-main.40.pdf