Vijay Devane
Author directory2026
BhashaBench V1: A Comprehensive Benchmark for the Quadrant of Indic Domains
Vijay Devane | Mohd Nauman | Bhargav Patel | Aniket Mahendra Wakchoure | Yogeshkumar Sant | Shyam Pawar | Viraj Thakur | Ananya Godse | Sunil Patra | Neha Maurya | Suraj Racha | Nitish Kamal Singh | Piyush Sawarkar | Kundeshwar Vijayrao Pundalik | Rohit Saluja | Ganesh Ramakrishnan
Findings of the Association for Computational Linguistics: EMNLP 2026
Vijay Devane | Mohd Nauman | Bhargav Patel | Aniket Mahendra Wakchoure | Yogeshkumar Sant | Shyam Pawar | Viraj Thakur | Ananya Godse | Sunil Patra | Neha Maurya | Suraj Racha | Nitish Kamal Singh | Piyush Sawarkar | Kundeshwar Vijayrao Pundalik | Rohit Saluja | Ganesh Ramakrishnan
Findings of the Association for Computational Linguistics: EMNLP 2026
The rapid advancement of large language models (LLMs) has intensified the need for domain and culture specific evaluation. Existing benchmarks are largely Anglocentric and domain-agnostic, limiting their applicability to India-centric contexts. To address this gap, we introduce BhashaBench V1, a large-scale, bilingual, exam-based benchmark covering four India-centric domains, Agriculture, Legal, Finance, and Ayurveda. BhashaBench V1 contains 74,166 meticulously curated question-answer pairs, with 52,494 in English and 21,672 in Hindi, sourced from authentic government and domain-specific exams. It is organized around a quadrant of four domains: Agriculture, Legal, Finance, and Ayurveda, comprising 90+ subdomains and covering 500+ topics, enabling fine-grained evaluation. Evaluation of 29+ LLMs reveals significant domain and language specific performance gaps, with especially large disparities in low-resource domains. For instance, GPT-4o achieves 76.49% overall accuracy in Legal but only 59.74% in Ayurveda. Models consistently perform better on English content compared to Hindi across all domains. Subdomain-level analysis shows that areas such as Cyber Law, International Finance perform relatively well, while Panchakarma, Seed Science, and Human Rights remain notably weak. BhashaBench V1 provides a large-scale, exam-sourced dataset for evaluating large language models across these four Indian knowledge domains. It enables assessment of models’ ability to integrate domain-specific knowledge with bilingual understanding. All code, benchmarks, and resources are publicly available to support open research.