Sanchari Chowdhuri
Author directory2026
IndicSafe: A Benchmark for Evaluating Multilingual LLM Safety in South Asia
Priyaranjan Pattnayak | Garima Panwar | Sanchari Chowdhuri
Findings of the Association for Computational Linguistics: EMNLP 2026
Priyaranjan Pattnayak | Garima Panwar | Sanchari Chowdhuri
Findings of the Association for Computational Linguistics: EMNLP 2026
As large language models (LLMs) are deployed in multilingual settings, their safety behavior in culturally diverse, low-resource languages remains poorly understood. We present the first systematic evaluation of LLM safety across 12 Indic languages, spoken by over 1.2 billion people but underrepresented in LLM training data. Using a dataset of 6,000 culturally grounded prompts spanning caste, religion, gender, health, and politics, we assess 18 leading LLMs on translated variants of the same prompt.
Our analysis reveals a previously underexamined failure mode: cross-language exact agreement is only 20.69%, indicating that identical prompts frequently receive different safety judgments across languages, while cross-model exact agreement is just 2.77%. Some models over-refuse or overflag politically sensitive content, whereas others fail to flag unsafe generations or shift failures into ambiguous outputs. We quantify these behaviors using prompt-level entropy, category bias scores, and multilingual consistency indices.
Our findings demonstrate critical safety generalization failures in multilingual LLMs and show that safety alignment does not reliably transfer across languages. We release IndicSafe, the first benchmark to enable culturally informed safety evaluation for Indic deployments, and advocate for language-aware alignment strategies grounded in regional harms.
Our analysis reveals a previously underexamined failure mode: cross-language exact agreement is only 20.69%, indicating that identical prompts frequently receive different safety judgments across languages, while cross-model exact agreement is just 2.77%. Some models over-refuse or overflag politically sensitive content, whereas others fail to flag unsafe generations or shift failures into ambiguous outputs. We quantify these behaviors using prompt-level entropy, category bias scores, and multilingual consistency indices.
Our findings demonstrate critical safety generalization failures in multilingual LLMs and show that safety alignment does not reliably transfer across languages. We release IndicSafe, the first benchmark to enable culturally informed safety evaluation for Indic deployments, and advocate for language-aware alignment strategies grounded in regional harms.
IndicJR: A Judge-Free Benchmark of Jailbreak Robustness in South Asian Languages
Priyaranjan Pattnayak | Sanchari Chowdhuri
Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track)
Priyaranjan Pattnayak | Sanchari Chowdhuri
Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track)
Safety alignment of large language models (LLMs) is mostly evaluated in English and contract-bound, leaving multilingual vulnerabilities understudied. We introduce Indic Jailbreak Robustness (IJR) a judge-free benchmark for adversarial safety across 12 Indic and South Asian languages (~2.09B speakers), covering 45,216 prompts in JSON (contract-bound) and Free (naturalistic) tracks.IJR reveals three patterns. (1) Contracts inflate refusals but do not stop jailbreaks: in JSON, LLaMA and Sarvam exceed 0.92 JSR, and in Free all models reach ~1.0 with refusals collapsing. (2) English→Indic attacks transfer strongly, with format wrappers often outperforming instruction wrappers. (3) Orthography matters: romanized/mixed inputs reduce JSR under JSON, with correlations to romanization share and tokenization ρ ≈ 0.28–0.32 indicating systematic effects. Human audits confirm detector reliability, and lite-to-full comparisons preserve conclusions. IJR offers a reproducible multilingual stress test revealing risks hidden by English-only, contract-focused evaluations, especially for South Asian users who frequently code-switch and romanize.
2025
LLM-Guided Lifecycle-Aware Clustering of Multi-Turn Customer Support Conversations
Priyaranjan Pattnayak | Sanchari Chowdhuri | Amit Agarwal | Hitesh Laxmichand Patel
Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics
Priyaranjan Pattnayak | Sanchari Chowdhuri | Amit Agarwal | Hitesh Laxmichand Patel
Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics
Clustering customer chat data is vital for cloud providers handling multi-service queries. Traditional methods struggle with overlapping concerns and create broad, static clusters that degrade over time. Re-clustering disrupts continuity, making issue tracking difficult. We propose an adaptive system that segments multi-turn chats into service-specific concerns and incrementally refines clusters as new issues arise. Cluster quality is tracked via Davies–Bouldin Index (DBI) and Silhouette Scores, with LLM-based splitting applied only to degraded clusters. Our method improves Silhouette Scores by over 100% and reduces DBI by 65.6% compared to baselines, enabling scalable, real-time analytics without full re-clustering.
Search