Swastik R
Author directory2026
Cross-Lingual Gaps in Visual STEM Reasoning: Auditing VLMs Across Six Indian Languages
Swastik R
Proceedings of the 6th Workshop on Multilingual Representation Learning (MRL 2026)
Swastik R
Proceedings of the 6th Workshop on Multilingual Representation Learning (MRL 2026)
Vision-language models (VLMs) are benchmarked on diagram-grounded science, technology, engineering, and mathematics (STEM) reasoning mostly in English, so it is unclear how much their accuracy changes when the same problem is posed in another language. We audit this for six Indian languages: 980 questions from MathVista, ScienceQA, and MMMU are machine-translated into Hindi, Bengali, Marathi, Tamil, Telugu, and Kannada with IndicTrans2 while the images stay unchanged. We evaluate seven open-weight VLMs (Qwen2.5-VL-7B/32B, Qwen3-VL-30B, InternVL2.5-8B, Aya-Vision- 8B, Gemma 3-27B, Llama-4-Maverick) and GPT-4o, plus no-image and chain-of-thought (CoT) ablations on Qwen2.5-VL-7B (68,600 inference records). Accuracy falls 9.8–24.9 percentage points (pp) from English to the Indian-language mean, and every model loses more on the Dravidian than on the Indo-Aryan languages (by 0.1–13.2 pp). For Qwen2.5- VL-7B with our prompt, CoT lowers Bengali (−14.4 pp) and Kannada (−11.4 pp) accuracy while barely changing English, but these losses coincide with more answer-extraction failures. Neither Aya-Vision-8B, trained for 23 languages of which only Hindi is among ours, nor scaling Qwen2.5-VL from 7B to 32B closes the gap, and response-level diagnostics show that fluent target-language output and correct answers can come apart. We quantify how machine translation, answer extraction, and English text in the images bound these measurements. Data and code are publicly available.