Co-attention based Multimodal Factorized Bilinear Pooling for Internet Memes Analysis

Gitanjali Kumari, Amitava Das, Asif Ekbal


Abstract
Social media platforms like Facebook, Twitter, and Instagram have a significant impact on several aspects of society. Memes are a new type of social media communication found on social platforms. Even though memes are primarily used to distribute humorous content, certain memes propagate hate speech through dark humor. It is critical to properly analyze and filter out these toxic memes from social media. But the presence of sarcasm and humor in an implicit way analyzes memes more challenging. This paper proposes an end-to-end neural network architecture that learns the complex association between text and image of a meme. For this purpose, we use a recent SemEval-2020 Task-8 multimodal dataset. We proposed an end-to-end CNN-based deep neural network architecture with two sub-modules viz. (i)Co-attention based sub-module and (ii) Multimodal Factorized Bilinear Pooling(MFB) sub-module to represent the textual and visual features of a meme in a more fine-grained way. We demonstrated the effectiveness of our proposed work through extensive experiments. The experimental results show that our proposed model achieves a 36.81% macro F1-score, outperforming all the baseline models.
Anthology ID:
2021.icon-main.31
Volume:
Proceedings of the 18th International Conference on Natural Language Processing (ICON)
Month:
December
Year:
2021
Address:
National Institute of Technology Silchar, Silchar, India
Editors:
Sivaji Bandyopadhyay, Sobha Lalitha Devi, Pushpak Bhattacharyya
Venue:
ICON
SIG:
Publisher:
NLP Association of India (NLPAI)
Note:
Pages:
261–270
Language:
URL:
https://aclanthology.org/2021.icon-main.31
DOI:
Bibkey:
Cite (ACL):
Gitanjali Kumari, Amitava Das, and Asif Ekbal. 2021. Co-attention based Multimodal Factorized Bilinear Pooling for Internet Memes Analysis. In Proceedings of the 18th International Conference on Natural Language Processing (ICON), pages 261–270, National Institute of Technology Silchar, Silchar, India. NLP Association of India (NLPAI).
Cite (Informal):
Co-attention based Multimodal Factorized Bilinear Pooling for Internet Memes Analysis (Kumari et al., ICON 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.icon-main.31.pdf
Data
SemEval-2020 Task-8