Multimodal Sentiment Detection Based on Multi-channel Graph Neural Networks

Xiaocui Yang, Shi Feng, Yifei Zhang, Daling Wang


Abstract
With the popularity of smartphones, we have witnessed the rapid proliferation of multimodal posts on various social media platforms. We observe that the multimodal sentiment expression has specific global characteristics, such as the interdependencies of objects or scenes within the image. However, most previous studies only considered the representation of a single image-text post and failed to capture the global co-occurrence characteristics of the dataset. In this paper, we propose Multi-channel Graph Neural Networks with Sentiment-awareness (MGNNS) for image-text sentiment detection. Specifically, we first encode different modalities to capture hidden representations. Then, we introduce multi-channel graph neural networks to learn multimodal representations based on the global characteristics of the dataset. Finally, we implement multimodal in-depth fusion with the multi-head attention mechanism to predict the sentiment of image-text pairs. Extensive experiments conducted on three publicly available datasets demonstrate the effectiveness of our approach for multimodal sentiment detection.
Anthology ID:
2021.acl-long.28
Volume:
Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)
Month:
August
Year:
2021
Address:
Online
Venues:
ACL | IJCNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
328–339
Language:
URL:
https://aclanthology.org/2021.acl-long.28
DOI:
10.18653/v1/2021.acl-long.28
Bibkey:
Cite (ACL):
Xiaocui Yang, Shi Feng, Yifei Zhang, and Daling Wang. 2021. Multimodal Sentiment Detection Based on Multi-channel Graph Neural Networks. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 328–339, Online. Association for Computational Linguistics.
Cite (Informal):
Multimodal Sentiment Detection Based on Multi-channel Graph Neural Networks (Yang et al., ACL 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.acl-long.28.pdf
Video:
 https://aclanthology.org/2021.acl-long.28.mp4
Code
 yangxiaocui1215/mgnns