Zheyuan Ryan Shi
Author directory2026
Script Gap: Evaluating LLM Triage on Indian Languages in Native vs Romanized Scripts in a Real-World Setting
Manurag Khullar | Utkarsh Desai | Poorva Malviya | Aman Dalmia | Zheyuan Ryan Shi
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: Industry Track
Manurag Khullar | Utkarsh Desai | Poorva Malviya | Aman Dalmia | Zheyuan Ryan Shi
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: Industry Track
Large Language Models (LLMs) are increasingly deployed in high-stakes clinical applications in India. Speakers of Indian languages frequently communicate using romanized text rather than native scripts, yet existing research rarely quantifies or evaluates this orthographic variation in real-world applications. We investigate how romanization impacts the reliability of LLMs in a critical domain: maternal and newborn healthcare triage. We benchmark leading LLMs on a real-world dataset of user-generated health queries spanning five Indian languages and Nepali. Our results reveal consistent degradation in performance for romanized messages, with gap reaching up to 24 points across languages and models. We propose and evaluate an Uncertainty-based Selective Routing method to narrow this script gap. Our findings highlight a critical safety blind spot in deploying LLMs for health systems: models that appear to understand romanized input may still fail to act on it reliably.
2025
Resolving UnderEdit & OverEdit with Iterative & Neighbor-Assisted Model Editing
Bhiman Kumar Baghel | Emma Jordan | Zheyuan Ryan Shi | Xiang Lorraine Li
Findings of the Association for Computational Linguistics: EMNLP 2025
Bhiman Kumar Baghel | Emma Jordan | Zheyuan Ryan Shi | Xiang Lorraine Li
Findings of the Association for Computational Linguistics: EMNLP 2025
Large Language Models (LLMs) are widely deployed in downstream tasks, but keeping their knowledge up-to-date via retraining or fine-tuning is often computationally expensive. Model editing provides a more efficient alternative by updating a targeted subset of parameters, which often follows the locate-and-edit paradigm. Despite this efficiency, existing methods are limited: edits may fail to inject knowledge (UnderEdit) or unintentionally disrupt unrelated neighboring knowledge (OverEdit). To address these challenges, we propose two complementary methods: iterative model editing, which applies successive edits to mitigate UnderEdit, and neighbor-assisted model editing, which incorporates neighboring knowledge during editing to reduce OverEdit. Our extensive experiments show that these techniques improve editing performance across multiple LLMs, algorithms, and benchmarks, reducing UnderEdit by up to 38 percentage points and OverEdit by up to 6, while remaining broadly applicable to any locate-and-edit method.