Harm or Humor: A Multimodal, Multilingual Benchmark for Overt and Covert Harmful Humor

Ahmed Sharshar, Hosam Elgendy, Yasser Rohaim, Saadeldine Eletter, Yuxia Wang


Abstract
Dark humor often relies on subtle cultural nuances and implicit cues that require contextual reasoning to interpret, posing safety challenges that current static benchmarks fail to capture. To address this, we introduce a novel multimodal, multilingual benchmark for detecting and understanding harmful and offensive humor. Our manually curated dataset comprises 3,000 texts and 6,000 images in English and Arabic, along with 1,200 videos spanning English, Arabic, and language-independent (universal) contexts. Unlike standard toxicity datasets, we enforce a strict annotation guideline: distinguishing Safe jokes from Harmful ones, with the latter further classified into Explicit (overt) and Implicit (covert) categories to probe deep reasoning. We systematically evaluate state-of-the-art (SOTA) open- and closed-source models across all modalities. Our findings reveal that closed-source models significantly outperform open-source models, with a notable performance difference between English and Arabic in both, underscoring the critical need for culturally grounded, reasoning-aware safety alignment. The code and dataset are available at https://github.com/insait-institute/harm-or-humor.

Content warning: This paper contains offensive text and meme imagery, video clips, audio, and other audiovisual material that may include sexual, violent, discriminatory, religious, disability-related, or historically sensitive material. Reader discretion is advised.
Anthology ID:
2026.emnlp-main.115
Volume:
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Month:
October
Year:
2026
Address:
Budapest, Hungary
Editors:
Hung-Yi Lee, Sunipa Dev, Jessy Li, Desmond Elliott
Venue:
EMNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
2680–2698
Language:
URL:
https://aclanthology.org/2026.emnlp-main.115/
DOI:
Bibkey:
Cite (ACL):
Ahmed Sharshar, Hosam Elgendy, Yasser Rohaim, Saadeldine Eletter, and Yuxia Wang. 2026. Harm or Humor: A Multimodal, Multilingual Benchmark for Overt and Covert Harmful Humor. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 2680–2698, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):
Harm or Humor: A Multimodal, Multilingual Benchmark for Overt and Covert Harmful Humor (Sharshar et al., EMNLP 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.emnlp-main.115.pdf