Safety evaluation of Large Language Models (LLMs) has largely focused on high-resource languages, leaving low-resource languages critically underserved. We present AlbanianLLMSafety, the first publicly available safety evaluation dataset for LLMs in Albanian, a linguistically distinct low-resource language with approximately 7.5 million speakers across Albania, Kosovo, North Macedonia, and the diaspora. The dataset contains 2,951 prompts spanning 11 safety categories, including self-harm, violence, racist content, child exploitation, and radicalization, with an average of 268 prompts per category. Each prompt is provided in Albanian with an English reference translation and a detailed category label. This resource addresses a significant gap in safety evaluation infrastructure for low-resource languages and provides an essential benchmark for developing safer, more inclusive LLMs. The dataset will be provided upon request to support safety evaluation, fine-tuning, red-teaming, and guardrail development for Albanian-speaking communities.

When Large Language Models (LLMs) are deployed in Chinese-language settings, a troubling pattern emerges: safety systems that work well in English break down. These systems struggle to cross linguistic and cultural boundaries, leaving models exposed to adversarial prompts that exploit Chinese-specific evasion techniques, including Pinyin romanization, character decomposition, Internet slang, and hedging tone. To address this gap, we introduce ChiSafe-PAS (Chinese Safety Pilot Annotation Set), a human-annotated benchmark of 1,897 adversarial Chinese prompts spanning four high-stakes domains: self-harm and violence, drugs and illicit trade, fraud, and satire. Of these, 1,544 entries carry complete gold-standard annotations: a 3-class response label (refuse, redirect, respond), a nine-category obfuscation taxonomy, a risk-level rating, and annotator rationale. We describe the dataset design, annotation process, and obfuscation taxonomy in detail. Our primary goal is practical: to give the research community a high-quality, culturally grounded resource for benchmarking LLM safety alignment. In doing so, we engage three broader tensions in the field: the blurring boundary between training and evaluation data, the need for domain coverage grounded in real-world risk, and the limits of scale as a substitute for cultural expertise.

We present StanceNakba 2026, a shared task on stance detection in polarized social media discourse related to the Palestinian-Israeli conflict, organized as part of Nakba-NLP 2026 at LREC-COLING 2026. The task introduces two subtasks: Subtask A (Actor-Level Stance Detection), which classifies English social media posts as Pro-Palestine, Pro-Israel, or Neutral; and Subtask B (Cross-Topic Stance Detection), which identifies Favor, Against, or Neither stances in Arabic posts toward two conflict-related topics, normalization with Israel and refugee presence in Jordan. The task is grounded in an annotated dataset of 2,606 social media posts. A total of 7 teams participated in Subtask A, and 6 teams in Subtask B. Participating systems primarily fine-tuned Arabic and multilingual transformer-based models, including MARBERT, AraBERT, and DeBERTa-v3 variants, with several teams employing cross-validation, ensemble methods, and topic-conditioned architectures. The best-performing systems achieved a Macro F1 of 0.9620 on Subtask A and 0.8724 on Subtask B, demonstrating that transformer-based approaches are highly effective for conflict-domain stance detection while highlighting persistent challenges in cross-topic generalization and neutral class prediction

We present ArabicDialectSafety, a human-curated Arabic safety dataset of 25,071 prompts covering six Arabic varieties—Modern Standard Arabic, Syrian, Egyptian, Algerian, Palestinian, and Moroccan—annotated with dialect labels and seven fine-grained harm categories. We introduce a dual-task evaluation framework for binary safe/unsafe detection and granular harm classification across dialects. Benchmarking seven supervised and generative models, we find that fine-tuned MARBERTv2 achieves the strongest performance (0.95 binary and 0.90 granular Macro-F1), substantially outperforming prompted frontier LLMs, including Arabic-specialized models. Our analyses show that dialect conditioning is most effective when integrated at the representation level, while significant performance gaps remain for low-resource Maghrebi dialects. We further evaluate seven frontier LLMs as response generators on harmful dialectal Arabic prompts and observe unsafe generation rates below 5% across models. We release the dataset and code upon acceptance to support future research on dialect-aware Arabic safety evaluation. Warning: this paper contains examples of harmful and potentially offensive content included solely for research purposes.

We introduce QatarGuard, an Arabic safety evaluation dataset containing 3,289 human curated prompts across eleven categories: Self-Harm, Violence, Child Exploitation, Vulgar Language, Radicalization, Sexual Language, Racist Content, Regulated Goods and Illegal Activities, Education, Family, and Health. We evaluate how safely five large language models respond to prompts: 299 from the Self-Harm category and 450 from three additional categories: Radicalization, Sexual Language, and Violence. The models tested include four commercial models (Claude 4.5 Sonnet, GPT-4.1, GPT-3.5 Turbo, and Google Gemini 2.5 Flash Lite) and one open-source Arabic-centric model (Fanar-2-27B-Instruct). We use GPT-5 as an automated judge, supported by manual review, to identify failure modes specific to Qatari Arabic dialect. For Self-Harm, unsafe response rates ranged from 2.34% (Gemini 2.5 Flash Lite) to 6.02% (GPT-3.5 Turbo), with Fanar-2-27B-Instruct at 4.35%. Across the other three categories, results varied more widely, from 2.91% to 13.48%. Sexual Language was the most challenging category, where even the best-performing model gave unsafe responses to 5.3% of prompts. Our findings show that Arabic dialect influences model behavior in safety-sensitive contexts, and that overall safety scores can hide weaknesses in specific categories. This highlights the need for safety evaluations that account for both dialect and category. The dataset is available for non-commercial research upon request, and evaluation scripts will be publicly released upon acceptance.