How Much Training Data Is Enough? Fine-Tuning LLMs for Short-Answer Scoring

Joshua A McGrane, David Torres Irribarra


Abstract
Fine-tuned models approached state-of-the-art agreement using small labelled sets. Open-weight models met operational criteria on seven of ten items, with a median of 50 responses per score point among passing items. A single marking exercise may supply enough data for automated short-answer scoring.
Anthology ID:
2026.aimecon-wip.39
Volume:
Proceedings of the Artificial Intelligence in Measurement and Education Conference (AIME-Con): Works in Progress
Month:
October
Year:
2026
Address:
Wyndham Grand Pittsburgh Downtown, Pittsburgh, Pennsylvania, United States
Editors:
Joshua Wilson, Christopher Ormerod, Magdalen Beiting-Parrish
Venue:
AIME-Con
SIG:
Publisher:
National Council on Measurement in Education (NCME)
Note:
Pages:
306–314
Language:
URL:
https://aclanthology.org/2026.aimecon-wip.39/
DOI:
Bibkey:
Cite (ACL):
Joshua A McGrane and David Torres Irribarra. 2026. How Much Training Data Is Enough? Fine-Tuning LLMs for Short-Answer Scoring. In Proceedings of the Artificial Intelligence in Measurement and Education Conference (AIME-Con): Works in Progress, pages 306–314, Wyndham Grand Pittsburgh Downtown, Pittsburgh, Pennsylvania, United States. National Council on Measurement in Education (NCME).
Cite (Informal):
How Much Training Data Is Enough? Fine-Tuning LLMs for Short-Answer Scoring (McGrane & Torres Irribarra, AIME-Con 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.aimecon-wip.39.pdf