Naga Sai Ritvik Modumudi
Author directory2026
IndicWinoBias: Measuring Occupational Gender Bias and Error Asymmetry in Indian Languages
Naga Sai Ritvik Modumudi | Dipti Sharma | Parameswari Krishnamurthy
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Naga Sai Ritvik Modumudi | Dipti Sharma | Parameswari Krishnamurthy
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
As Large Language Models (LLMs) are increasingly deployed in multilingual contexts, evaluating their fairness across languages is essential. We investigate occupational gender bias in 16 LLMs across eight Indic languages using a controlled coreference resolution benchmark that employs transliterated English occupational loanwords to minimize grammatical gender cues. In addition to this generative evaluation, we compute intrinsic bias metrics (WEAT and DirectBias) from the models’ embedding spaces. Across models, we observe systematic disparities between pro-stereotype and anti-stereotype conditions. These gaps are largest in high-accuracy models, whereas models lacking baseline competence exhibit smaller gaps driven by general prediction uncertainty. Error analysis shows a consistent asymmetry, with higher error rates when the correct referent is female. However, intrinsic embedding bias metrics show no statistically significant correlation with these generative disparities (p \gt 0.05). These results suggest that strong linguistic competence does not necessarily correspond to stereotype-neutral generation and that intrinsic embedding metrics alone may not reliably predict bias in multilingual language model behavior.