Mitigating Inconsistencies in Multimodal Sentiment Analysis under Uncertain Missing Modalities

Jiandian Zeng, Jiantao Zhou, Tianyi Liu


Abstract
For the missing modality problem in Multimodal Sentiment Analysis (MSA), the inconsistency phenomenon occurs when the sentiment changes due to the absence of a modality. The absent modality that determines the overall semantic can be considered as a key missing modality. However, previous works all ignored the inconsistency phenomenon, simply discarding missing modalities or solely generating associated features from available modalities. The neglect of the key missing modality case may lead to incorrect semantic results. To tackle the issue, we propose an Ensemble-based Missing Modality Reconstruction (EMMR) network to detect and recover semantic features of the key missing modality. Specifically, we first learn joint representations with remaining modalities via a backbone encoder-decoder network. Then, based on the recovered features, we check the semantic consistency to determine whether the absent modality is crucial to the overall sentiment polarity. Once the inconsistency problem due to the key missing modality exists, we integrate several encoder-decoder approaches for better decision making. Extensive experiments and analyses are conducted on CMU-MOSI and IEMOCAP datasets, validating the superiority of the proposed method.
Anthology ID:
2022.emnlp-main.189
Volume:
Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
Month:
December
Year:
2022
Address:
Abu Dhabi, United Arab Emirates
Editors:
Yoav Goldberg, Zornitsa Kozareva, Yue Zhang
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2924–2934
Language:
URL:
https://aclanthology.org/2022.emnlp-main.189
DOI:
10.18653/v1/2022.emnlp-main.189
Bibkey:
Cite (ACL):
Jiandian Zeng, Jiantao Zhou, and Tianyi Liu. 2022. Mitigating Inconsistencies in Multimodal Sentiment Analysis under Uncertain Missing Modalities. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2924–2934, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Cite (Informal):
Mitigating Inconsistencies in Multimodal Sentiment Analysis under Uncertain Missing Modalities (Zeng et al., EMNLP 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.emnlp-main.189.pdf