SESAM at SemEval-2020 Task 8: Investigating the Relationship between Image and Text in Sentiment Analysis of Memes

Lisa Bonheme, Marek Grzes


Abstract
This paper presents our submission to task 8 (memotion analysis) of the SemEval 2020 competition. We explain the algorithms that were used to learn our models along with the process of tuning the algorithms and selecting the best model. Since meme analysis is a challenging task with two distinct modalities, we studied the impact of different multimodal representation strategies. The results of several approaches to dealing with multimodal data are therefore discussed in the paper. We found that alignment-based strategies did not perform well on memes. Our quantitative results also showed that images and text were uncorrelated. Fusion-based strategies did not show significant improvements and using one modality only (text or image) tends to lead to better results when applied with the predictive models that we used in our research.
Anthology ID:
2020.semeval-1.102
Volume:
Proceedings of the Fourteenth Workshop on Semantic Evaluation
Month:
December
Year:
2020
Address:
Barcelona (online)
Venue:
SemEval
SIGs:
SIGLEX | SIGSEM
Publisher:
International Committee for Computational Linguistics
Note:
Pages:
804–816
Language:
URL:
https://aclanthology.org/2020.semeval-1.102
DOI:
10.18653/v1/2020.semeval-1.102
Award:
 Best Paper Honorable Mention
Bibkey:
Cite (ACL):
Lisa Bonheme and Marek Grzes. 2020. SESAM at SemEval-2020 Task 8: Investigating the Relationship between Image and Text in Sentiment Analysis of Memes. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 804–816, Barcelona (online). International Committee for Computational Linguistics.
Cite (Informal):
SESAM at SemEval-2020 Task 8: Investigating the Relationship between Image and Text in Sentiment Analysis of Memes (Bonheme & Grzes, SemEval 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.semeval-1.102.pdf