Aman Dalmia
Author directory2026
Script Gap: Evaluating LLM Triage on Indian Languages in Native vs Romanized Scripts in a Real-World Setting
Manurag Khullar | Utkarsh Desai | Poorva Malviya | Aman Dalmia | Zheyuan Ryan Shi
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: Industry Track
Manurag Khullar | Utkarsh Desai | Poorva Malviya | Aman Dalmia | Zheyuan Ryan Shi
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: Industry Track
Large Language Models (LLMs) are increasingly deployed in high-stakes clinical applications in India. Speakers of Indian languages frequently communicate using romanized text rather than native scripts, yet existing research rarely quantifies or evaluates this orthographic variation in real-world applications. We investigate how romanization impacts the reliability of LLMs in a critical domain: maternal and newborn healthcare triage. We benchmark leading LLMs on a real-world dataset of user-generated health queries spanning five Indian languages and Nepali. Our results reveal consistent degradation in performance for romanized messages, with gap reaching up to 24 points across languages and models. We propose and evaluate an Uncertainty-based Selective Routing method to narrow this script gap. Our findings highlight a critical safety blind spot in deploying LLMs for health systems: models that appear to understand romanized input may still fail to act on it reliably.
2025
GeoCoder: Solving Geometry Problems by Generating Modular Code through Vision-Language Models
Aditya Sharma | Aman Dalmia | Mehran Kazemi | Amal Zouaq | Christopher Pal
Findings of the Association for Computational Linguistics: NAACL 2025
Aditya Sharma | Aman Dalmia | Mehran Kazemi | Amal Zouaq | Christopher Pal
Findings of the Association for Computational Linguistics: NAACL 2025
Geometry problem-solving demands advanced reasoning abilities to process multimodal inputs and employ mathematical knowledge effectively. Vision-language models (VLMs) have made significant progress in various multimodal tasks. Yet, they still struggle with geometry problems and are significantly limited by their inability to perform mathematical operations not seen during pre-training, such as calculating the cosine of an arbitrary angle, and by difficulties in correctly applying relevant geometry formulas. To overcome these challenges, we present GeoCoder, which leverages modular code-finetuning to generate and execute code using a predefined geometry function library. By executing the code, we achieve accurate and deterministic calculations, contrasting the stochastic nature of autoregressive token prediction, while the function library minimizes errors in formula usage. We also propose a multimodal retrieval-augmented variant of GeoCoder, named RAG-GeoCoder, which incorporates a non-parametric memory module for retrieving functions from the geometry library, thereby reducing reliance on parametric memory. Our modular code-finetuning approach enhances the geometric reasoning capabilities of VLMs, yielding an average improvement of over 16% across various question complexities on the GeomVerse dataset compared to other fine-tuning methods.