Gihyeon Choi

Author directory

2026

Evaluating LLMs as math tutors is challenging because many automated protocols either score a single response against a reference or place the model in a fixed dialogue context, without measuring its ability to build and steer tutoring interactions. We present a fully automated, simulation-based framework that generates controlled multi-turn teacher–student dialogues with turn-level answer verification under a fixed interaction budget. This budget specifies the maximum number of student–teacher rounds allowed per problem (K=5 in our main experiments). To prevent score inflation from answer-revealing feedback, we introduce an overinformative eligibility filter: a hint-only student test applied to the teacher utterance immediately preceding the student’s first correct attempt. Only eligible corrections receive credit in our effectiveness and efficiency metrics, both of which are normalized by the number of initially incorrect sessions. We also propose a diagnostic metric suite separating tutoring effectiveness/efficiency (ECR/FEI), answer reachability (ARG), and teacher judgment (correctness/incorrectness F1). Experiments on GSM8K and MATH with diverse teacher and student models show that problem-solving accuracy is an informative but incomplete proxy for interactive tutoring ability, and that filtering overinformative feedback can substantially alter apparent performance. Our framework enables scalable, reproducible, and fine-grained evaluation of LLM math tutoring beyond outcome-only scoring.

2025

This paper presents automatic evaluation systems for assessing the pedagogical capabilities of LLM-based AI tutors. Drawing from a shared task, our systems specifically target four key dimensions of tutor responses: Mistake Identification, Mistake Location, Providing Guidance, and Actionability. These dimensions capture the educational quality of responses from multiple perspectives, including the ability to detect student mistakes, accurately identify error locations, provide effective instructional guidance, and offer actionable feedback. We propose GPT-4.1-based automatic evaluation systems, leveraging their strong capabilities in comprehending diverse linguistic expressions and complex conversational contexts to address the detailed evaluation criteria across these dimensions. Our systems were quantitatively evaluated based on the official criteria of each track. In the Mistake Location track, our evaluation systems achieved an Exact macro F1 score of 58.80% (ranked in the top 3), and in the Providing Guidance track, they achieved 56.06% (ranked in the top 5). While the systems showed mid-range performance in the remaining tracks, the overall results demonstrate that our proposed automatic evaluation systems can effectively assess the quality of tutor responses, highlighting their potential for evaluating AI tutor effectiveness.
Search