Self-Revising Agents as Item Writers: Benchmarking Against Professional Item Writing

Steven Tang, Zhen Li, Richard Patz


Abstract
Expert reviewers rated 156 calculus items from a self-revising multi-agent framework (Claude Sonnet 4.6 or GPT-5.4, three prompt conditions) and 26 human-written items. AI items were formative-ready nearly as often as human items (78–80% versus 88%) but summative-ready less often (26% and 10% versus 58%); a root-item reference mattered most.
Anthology ID:
2026.aimecon-wip.47
Volume:
Proceedings of the Artificial Intelligence in Measurement and Education Conference (AIME-Con): Works in Progress
Month:
October
Year:
2026
Address:
Wyndham Grand Pittsburgh Downtown, Pittsburgh, Pennsylvania, United States
Editors:
Joshua Wilson, Christopher Ormerod, Magdalen Beiting-Parrish
Venue:
AIME-Con
SIG:
Publisher:
National Council on Measurement in Education (NCME)
Note:
Pages:
369–376
Language:
URL:
https://aclanthology.org/2026.aimecon-wip.47/
DOI:
Bibkey:
Cite (ACL):
Steven Tang, Zhen Li, and Richard Patz. 2026. Self-Revising Agents as Item Writers: Benchmarking Against Professional Item Writing. In Proceedings of the Artificial Intelligence in Measurement and Education Conference (AIME-Con): Works in Progress, pages 369–376, Wyndham Grand Pittsburgh Downtown, Pittsburgh, Pennsylvania, United States. National Council on Measurement in Education (NCME).
Cite (Informal):
Self-Revising Agents as Item Writers: Benchmarking Against Professional Item Writing (Tang et al., AIME-Con 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.aimecon-wip.47.pdf