Fine-tuning and Sampling Strategies for Multimodal Role Labeling of Entities under Class Imbalance

Syrielle Montariol, Étienne Simon, Arij Riabi, Djamé Seddah


Abstract
We propose our solution to the multimodal semantic role labeling task from the CONSTRAINT’22 workshop. The task aims at classifying entities in memes into classes such as “hero” and “villain”. We use several pre-trained multi-modal models to jointly encode the text and image of the memes, and implement three systems to classify the role of the entities. We propose dynamic sampling strategies to tackle the issue of class imbalance. Finally, we perform qualitative analysis on the representations of the entities.
Anthology ID:
2022.constraint-1.7
Volume:
Proceedings of the Workshop on Combating Online Hostile Posts in Regional Languages during Emergency Situations
Month:
May
Year:
2022
Address:
Dublin, Ireland
Editors:
Tanmoy Chakraborty, Md. Shad Akhtar, Kai Shu, H. Russell Bernard, Maria Liakata, Preslav Nakov, Aseem Srivastava
Venue:
CONSTRAINT
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
55–65
Language:
URL:
https://aclanthology.org/2022.constraint-1.7
DOI:
10.18653/v1/2022.constraint-1.7
Bibkey:
Cite (ACL):
Syrielle Montariol, Étienne Simon, Arij Riabi, and Djamé Seddah. 2022. Fine-tuning and Sampling Strategies for Multimodal Role Labeling of Entities under Class Imbalance. In Proceedings of the Workshop on Combating Online Hostile Posts in Regional Languages during Emergency Situations, pages 55–65, Dublin, Ireland. Association for Computational Linguistics.
Cite (Informal):
Fine-tuning and Sampling Strategies for Multimodal Role Labeling of Entities under Class Imbalance (Montariol et al., CONSTRAINT 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.constraint-1.7.pdf
Video:
 https://aclanthology.org/2022.constraint-1.7.mp4