Evaluating an AI-Item Generation Tool for NAEP Science

Carolina Safar, Mitchell Price, Jeffery Ackley


Abstract
AI-enabled systems could transform item authoring for standardized assessments. We compare framework alignment and accuracy of NAEP Science items developed by two AI tools and human authors. Items from all sources had similar acceptance rates and most accepted items would require significant revision, underscoring the importance of evaluating AI-generated content.
Anthology ID:
2026.aimecon-sessions.21
Volume:
Proceedings of the Artificial Intelligence in Measurement and Education Conference (AIME-Con): Coordinated Session Papers
Month:
October
Year:
2026
Address:
Wyndham Grand Pittsburgh Downtown, Pittsburgh, Pennsylvania, United States
Editors:
Joshua Wilson, Christopher Ormerod, Magdalen Beiting-Parrish
Venue:
AIME-Con
SIG:
Publisher:
National Council on Measurement in Education (NCME)
Note:
Pages:
203–209
Language:
URL:
https://aclanthology.org/2026.aimecon-sessions.21/
DOI:
Bibkey:
Cite (ACL):
Carolina Safar, Mitchell Price, and Jeffery Ackley. 2026. Evaluating an AI-Item Generation Tool for NAEP Science. In Proceedings of the Artificial Intelligence in Measurement and Education Conference (AIME-Con): Coordinated Session Papers, pages 203–209, Wyndham Grand Pittsburgh Downtown, Pittsburgh, Pennsylvania, United States. National Council on Measurement in Education (NCME).
Cite (Informal):
Evaluating an AI-Item Generation Tool for NAEP Science (Safar et al., AIME-Con 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.aimecon-sessions.21.pdf