Subhajit Roy

Author directory

2026

Large Language Models (LLMs) are increasingly used by students as informal assistants for problem solving and concept clarification. Despite growing adoption, the multilingual reliability of these systems for educational mathematics remains under-explored, particularly across Indian regional languages. In this work, we present a multilingual evaluation of contemporary LLMs on high-school mathematics problems across six Indian languages: English, Hindi, Bengali, Urdu, Gujarati, and Malayalam. To guide the design of lightweight prompting conditions, we conduct a small exploratory prompt elicitation exercise with Grade 11–12 students and observe that students predominantly use concise direct-solution requests and lightweight expertise-oriented phrasing. Guided by these observations, we evaluate GPT-4o-mini, LLaMA 3-8B, and DeepSeek-R1 using simplified student-inspired prompting strategies on a parallel corpus of numerically perturbed textbook mathematics problems. Our results reveal consistent multilingual performance disparities, with accuracy strongest in English and lower across several regional languages. We further analyze the effects of prompting strategy and mathematical topic, and provide qualitative error analysis identifying recurring arithmetic, reasoning, and regional-language comprehension failures. These findings highlight the importance of evaluating multilingual educational reliability as LLM-based learning assistance becomes increasingly accessible across diverse linguistic settings.

2025

There are serious attempts at improving the mathematical acumen of LLMs in questions posed in English. In India, where a large fraction of the students study in regional languages, there is a need to assess and improve these state-of-the-art LLMs in their reasoning abilities in regional languages as well. As Hindi is a language predominantly used in India, this study proposes a new dataset on mathematical combinatorics problems consisting of a parallel corpus of problems in English and Hindi collected from NCERT textbooks. We evaluate the “raw” single-shot capabilities of these LLMs in solving problems posed in Hindi. Then we apply a chain-of-thought approach to evaluate the improvement in the abilities of the LLMs at solving combinatorics problems posed in Hindi. Our study reveals that while smaller LLMs like LLaMa3-8B shows a significant drop in performance when questions are posed in Hindi, versus questions posed in English, larger LLMs like GPT4-turbo shows excellent capabilities at solving problems posed in Hindi, almost at par its abilities in English. We make two primary inferences from our study: (1) large models like GPT4 can be readily deployed in schools where Hindi is the primary language of study, especially in rural India; (2) there is a need to improve the multilingual capabilities of smaller models. As these smaller open-source models can be deployed on not so expensive GPUs, it is easier for schools to provide these models to the students, and hence, the latter is an important direction for future research.