Can LLMs Interpret Psychometric Parameters? Quantifying "Over-Knowledge" in LLM Student Simulation

Kazunori Fukuhara, Benjamin Domingue


Abstract
LLM simulation suffers from an “over-knowledge problem”, where models perform too well to represent struggling learners. We compare persona-, IRT-, and CDM-based prompting for student simulation and measure how well methods reflect expected behaviors and quantify this bias. Findings show LLMs follow IRT parameters, yet struggle to simulate real abilities.
Anthology ID:
2026.aimecon-main.47
Volume:
Proceedings of the Artificial Intelligence in Measurement and Education Conference (AIME-Con): Full Papers
Month:
October
Year:
2026
Address:
Wyndham Grand Pittsburgh Downtown, Pittsburgh, Pennsylvania, United States
Editors:
Joshua Wilson, Christopher Ormerod, Magdalen Beiting-Parrish
Venue:
AIME-Con
SIG:
Publisher:
National Council on Measurement in Education (NCME)
Note:
Pages:
421–432
Language:
URL:
https://aclanthology.org/2026.aimecon-main.47/
DOI:
Bibkey:
Cite (ACL):
Kazunori Fukuhara and Benjamin Domingue. 2026. Can LLMs Interpret Psychometric Parameters? Quantifying "Over-Knowledge" in LLM Student Simulation. In Proceedings of the Artificial Intelligence in Measurement and Education Conference (AIME-Con): Full Papers, pages 421–432, Wyndham Grand Pittsburgh Downtown, Pittsburgh, Pennsylvania, United States. National Council on Measurement in Education (NCME).
Cite (Informal):
Can LLMs Interpret Psychometric Parameters? Quantifying “Over-Knowledge” in LLM Student Simulation (Fukuhara & Domingue, AIME-Con 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.aimecon-main.47.pdf