Garima Panwar

Author directory

2026

As large language models (LLMs) are deployed in multilingual settings, their safety behavior in culturally diverse, low-resource languages remains poorly understood. We present the first systematic evaluation of LLM safety across 12 Indic languages, spoken by over 1.2 billion people but underrepresented in LLM training data. Using a dataset of 6,000 culturally grounded prompts spanning caste, religion, gender, health, and politics, we assess 18 leading LLMs on translated variants of the same prompt.

Our analysis reveals a previously underexamined failure mode: cross-language exact agreement is only 20.69%, indicating that identical prompts frequently receive different safety judgments across languages, while cross-model exact agreement is just 2.77%. Some models over-refuse or overflag politically sensitive content, whereas others fail to flag unsafe generations or shift failures into ambiguous outputs. We quantify these behaviors using prompt-level entropy, category bias scores, and multilingual consistency indices.

Our findings demonstrate critical safety generalization failures in multilingual LLMs and show that safety alignment does not reliably transfer across languages. We release IndicSafe, the first benchmark to enable culturally informed safety evaluation for Indic deployments, and advocate for language-aware alignment strategies grounded in regional harms.