%0 Conference Proceedings %T Memes in the Wild: Assessing the Generalizability of the Hateful Memes Challenge Dataset %A Kirk, Hannah %A Jun, Yennie %A Rauba, Paulius %A Wachtel, Gal %A Li, Ruining %A Bai, Xingjian %A Broestl, Noah %A Doff-Sotta, Martin %A Shtedritski, Aleksandar %A Asano, Yuki M. %Y Mostafazadeh Davani, Aida %Y Kiela, Douwe %Y Lambert, Mathias %Y Vidgen, Bertie %Y Prabhakaran, Vinodkumar %Y Waseem, Zeerak %S Proceedings of the 5th Workshop on Online Abuse and Harms (WOAH 2021) %D 2021 %8 August %I Association for Computational Linguistics %C Online %F kirk-etal-2021-memes %X Hateful memes pose a unique challenge for current machine learning systems because their message is derived from both text- and visual-modalities. To this effect, Facebook released the Hateful Memes Challenge, a dataset of memes with pre-extracted text captions, but it is unclear whether these synthetic examples generalize to ‘memes in the wild’. In this paper, we collect hateful and non-hateful memes from Pinterest to evaluate out-of-sample performance on models pre-trained on the Facebook dataset. We find that ‘memes in the wild’ differ in two key aspects: 1) Captions must be extracted via OCR, injecting noise and diminishing performance of multimodal models, and 2) Memes are more diverse than ‘traditional memes’, including screenshots of conversations or text on a plain background. This paper thus serves as a reality-check for the current benchmark of hateful meme detection and its applicability for detecting real world hate. %R 10.18653/v1/2021.woah-1.4 %U https://aclanthology.org/2021.woah-1.4 %U https://doi.org/10.18653/v1/2021.woah-1.4 %P 26-35