DD-TIG at Constraint@ACL2022: Multimodal Understanding and Reasoning for Role Labeling of Entities in Hateful Memes

Ziming Zhou, Han Zhao, Jingjing Dong, Jun Gao, Xiaolong Liu


Abstract
The memes serve as an important tool in online communication, whereas some hateful memes endanger cyberspace by attacking certain people or subjects. Recent studies address hateful memes detection while further understanding of relationships of entities in memes remains unexplored. This paper presents our work at the Constraint@ACL2022 Shared Task: Hero, Villain and Victim: Dissecting harmful memes for semantic role labelling of entities. In particular, we propose our approach utilizing transformer-based multimodal models through a VCR method with data augmentation, continual pretraining, loss re-weighting, and ensemble learning. We describe the models used, the ways of preprocessing and experiments implementation. As a result, our best model achieves the Macro F1-score of 54.707 on the test set of this shared task.
Anthology ID:
2022.constraint-1.2
Volume:
Proceedings of the Workshop on Combating Online Hostile Posts in Regional Languages during Emergency Situations
Month:
May
Year:
2022
Address:
Dublin, Ireland
Editors:
Tanmoy Chakraborty, Md. Shad Akhtar, Kai Shu, H. Russell Bernard, Maria Liakata, Preslav Nakov, Aseem Srivastava
Venue:
CONSTRAINT
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
12–18
Language:
URL:
https://aclanthology.org/2022.constraint-1.2
DOI:
10.18653/v1/2022.constraint-1.2
Bibkey:
Cite (ACL):
Ziming Zhou, Han Zhao, Jingjing Dong, Jun Gao, and Xiaolong Liu. 2022. DD-TIG at Constraint@ACL2022: Multimodal Understanding and Reasoning for Role Labeling of Entities in Hateful Memes. In Proceedings of the Workshop on Combating Online Hostile Posts in Regional Languages during Emergency Situations, pages 12–18, Dublin, Ireland. Association for Computational Linguistics.
Cite (Informal):
DD-TIG at Constraint@ACL2022: Multimodal Understanding and Reasoning for Role Labeling of Entities in Hateful Memes (Zhou et al., CONSTRAINT 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.constraint-1.2.pdf
Data
Hateful MemesVCR