Multi-Agent LLM Annotation and Scoring for Training Fine-Grained K-12 Writing-Feedback Models

Justin O Barber, Michael P Hemenway, Martha Bellows, Susan Lottridge


Abstract
Fine-grained formative writing feedback needs dense, standards-aligned labels that human annotation cannot supply at scale. We describe a multi-agent LLM pipeline producing verified silver labels, and then train small deterministic transformer scorers. On a Grade 5 pilot these reach Cohen’s 𝜅 up to 0.92 on conventions.
Anthology ID:
2026.aimecon-sessions.1
Volume:
Proceedings of the Artificial Intelligence in Measurement and Education Conference (AIME-Con): Coordinated Session Papers
Month:
October
Year:
2026
Address:
Wyndham Grand Pittsburgh Downtown, Pittsburgh, Pennsylvania, United States
Editors:
Joshua Wilson, Christopher Ormerod, Magdalen Beiting-Parrish
Venue:
AIME-Con
SIG:
Publisher:
National Council on Measurement in Education (NCME)
Note:
Pages:
1–8
Language:
URL:
https://aclanthology.org/2026.aimecon-sessions.1/
DOI:
Bibkey:
Cite (ACL):
Justin O Barber, Michael P Hemenway, Martha Bellows, and Susan Lottridge. 2026. Multi-Agent LLM Annotation and Scoring for Training Fine-Grained K-12 Writing-Feedback Models. In Proceedings of the Artificial Intelligence in Measurement and Education Conference (AIME-Con): Coordinated Session Papers, pages 1–8, Wyndham Grand Pittsburgh Downtown, Pittsburgh, Pennsylvania, United States. National Council on Measurement in Education (NCME).
Cite (Informal):
Multi-Agent LLM Annotation and Scoring for Training Fine-Grained K-12 Writing-Feedback Models (Barber et al., AIME-Con 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.aimecon-sessions.1.pdf