Rajan Das Gupta

Author directory

2026

Large Language Models (LLMs) have shown impressive capabilities across many language understanding tasks; however, their ability to perform numerical reasoning remains fragile, particularly outside English. This limitation is especially concerning for Hindi, a language with hundreds of millions of speakers but relatively little focused evaluation in numerically grounded reasoning. In this work, we present HI-REASO, a benchmark designed to systematically evaluate numerical reasoning in Hindi. The benchmark comprises six complementary task types and 3k instances drawn from educational materials, real-world problem formulations, and commonsense reasoning scenarios. We assess a range of state-of-the-art LLMs using Hindi-native prompting as well as cross-lingual prompting and cross-lingual chain-of-thought reasoning. Across all settings, models consistently underperform compared to human baselines, with the largest performance degradation observed when reasoning is conducted directly in Hindi. While cross-lingual strategies offer partial improvements, they do not close the gap. Our findings reveal fundamental shortcomings in current LLMs’ numerical reasoning abilities under Hindi-native conditions and motivate the development of models and training paradigms that better support linguistically diverse numerical reasoning. We publicly release the code and data at https://github.com/kraritt/hi-reaso.