Md Tanvirul Islam Turad
Author directory2026
Assessing Large Language Models as Automated Writing Evaluation and Feedback Tools: A Human-AI Agreement Study on L2 English Writing
Kowshik Saha | Moyen Uddin | Md Tanvirul Islam Turad | Mizanur Rahman
Proceedings of the 10th Widening NLP Workshop
Kowshik Saha | Moyen Uddin | Md Tanvirul Islam Turad | Mizanur Rahman
Proceedings of the 10th Widening NLP Workshop
Large language models (LLMs) are often suggested as tools for grading student writing and giving feedback, but it is unclear how far we can trust them on the separate skills that make up good writing. We compare two LLM graders (Gemini and Llama) with eight professional English teachers on 20 paragraphs written in second-language English by Bengali-speaking undergraduates in Bangladesh, each graded by two teachers, scoring content, organization, vocabulary, and grammar separately as well as the total. We then run a second stage in which teachers read each model’s written explanation, judge whether they agree with it, and rate the feedback. The LLMs match the teachers on the total score, content, and organization, but are unreliable on vocabulary and grammar, where the two models do not even agree with each other. We also show that one low agreement score is misleading: it comes from graders giving very similar marks (score clustering), not from real disagreement. In the second stage, teachers found the AI feedback safe and useful most of the time (≥ 77.5%), with very few made-up errors (2.5%), and preferred Gemini’s stricter grammar score to their own 20% of the time. LLMs can help grade writing, but a teacher should audit their output, with extra care on vocabulary and grammar.