Naga Sai Ritvik Modumudi

Author directory

2026

As Large Language Models (LLMs) are increasingly deployed in multilingual contexts, evaluating their fairness across languages is essential. We investigate occupational gender bias in 16 LLMs across eight Indic languages using a controlled coreference resolution benchmark that employs transliterated English occupational loanwords to minimize grammatical gender cues. In addition to this generative evaluation, we compute intrinsic bias metrics (WEAT and DirectBias) from the models’ embedding spaces. Across models, we observe systematic disparities between pro-stereotype and anti-stereotype conditions. These gaps are largest in high-accuracy models, whereas models lacking baseline competence exhibit smaller gaps driven by general prediction uncertainty. Error analysis shows a consistent asymmetry, with higher error rates when the correct referent is female. However, intrinsic embedding bias metrics show no statistically significant correlation with these generative disparities (p \gt 0.05). These results suggest that strong linguistic competence does not necessarily correspond to stereotype-neutral generation and that intrinsic embedding metrics alone may not reliably predict bias in multilingual language model behavior.