Mohammed Abdelkreem
Author directory2026
QatarGuard: A Culturally-Aware Safety Benchmark for Arabic and Multilingual LLMs
Wajdi Zaghouani | Kholoud Khalil Aldous | Md. Rafiul Biswas | Mohammed Abdelkreem
Findings of the Association for Computational Linguistics: EMNLP 2026
Wajdi Zaghouani | Kholoud Khalil Aldous | Md. Rafiul Biswas | Mohammed Abdelkreem
Findings of the Association for Computational Linguistics: EMNLP 2026
We introduce QatarGuard, an Arabic safety evaluation dataset containing 3,289 human curated prompts across eleven categories: Self-Harm, Violence, Child Exploitation, Vulgar Language, Radicalization, Sexual Language, Racist Content, Regulated Goods and Illegal Activities, Education, Family, and Health. We evaluate how safely five large language models respond to prompts: 299 from the Self-Harm category and 450 from three additional categories: Radicalization, Sexual Language, and Violence. The models tested include four commercial models (Claude 4.5 Sonnet, GPT-4.1, GPT-3.5 Turbo, and Google Gemini 2.5 Flash Lite) and one open-source Arabic-centric model (Fanar-2-27B-Instruct). We use GPT-5 as an automated judge, supported by manual review, to identify failure modes specific to Qatari Arabic dialect. For Self-Harm, unsafe response rates ranged from 2.34% (Gemini 2.5 Flash Lite) to 6.02% (GPT-3.5 Turbo), with Fanar-2-27B-Instruct at 4.35%. Across the other three categories, results varied more widely, from 2.91% to 13.48%. Sexual Language was the most challenging category, where even the best-performing model gave unsafe responses to 5.3% of prompts. Our findings show that Arabic dialect influences model behavior in safety-sensitive contexts, and that overall safety scores can hide weaknesses in specific categories. This highlights the need for safety evaluations that account for both dialect and category. The dataset is available for non-commercial research upon request, and evaluation scripts will be publicly released upon acceptance.