Fawzia Kara-Isitt

Author directory

2026

This paper investigates the near-total absence of Urdu from mainstream LLM safety evaluation and nine years of WOAH proceedings. A complete enumeration of all 205 papers across nine ALW/WOAH editions confirms zero dedicated Urdu papers across the entire period. To assess whether this absence has measurable consequences, five large language models (GPT-4o, Claude Sonnet, Gemini 2.5 Flash, Llama-3.1, and Qwen-2.5) were evaluated across six datasets spanning Nastaliq Urdu, Roman Urdu, English, and code-switched Urdu-English, producing approximately 28,000 classifications. Label instability between original-script and English-translation classification ranged from 15.9% to 31.6%, and a missed-in-Urdu rate content flagged as harmful in English but passed as normal in Urdu ranged from 2.4% to 9.9%. Pairwise significance testing confirms that frontier closed models form a statistically indistinguishable cluster significantly outperforming smaller open-weight models on both metrics. The paper also identifies the absence of a dedicated Nastaliq Urdu with English code-switched hate speech dataset as a structural gap, and raises questions about annotation bias in existing Urdu corpora. The findings demonstrate that Urdu speakers are exposed to measurably weaker content moderation safety assurance, a gap that this paper documents, quantifies, and calls on the research community to address, considering that it is in the top 10 most spoken and used language in the world.