Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities

Jinming Zhao, Ruichen Li, Qin Jin


Abstract
Multimodal fusion has been proved to improve emotion recognition performance in previous works. However, in real-world applications, we often encounter the problem of missing modality, and which modalities will be missing is uncertain. It makes the fixed multimodal fusion fail in such cases. In this work, we propose a unified model, Missing Modality Imagination Network (MMIN), to deal with the uncertain missing modality problem. MMIN learns robust joint multimodal representations, which can predict the representation of any missing modality given available modalities under different missing modality conditions. Comprehensive experiments on two benchmark datasets demonstrate that the unified MMIN model significantly improves emotion recognition performance under both uncertain missing-modality testing conditions and full-modality ideal testing condition. The code will be available at https://github.com/AIM3-RUC/MMIN.
Anthology ID:
2021.acl-long.203
Volume:
Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)
Month:
August
Year:
2021
Address:
Online
Editors:
Chengqing Zong, Fei Xia, Wenjie Li, Roberto Navigli
Venues:
ACL | IJCNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2608–2618
Language:
URL:
https://aclanthology.org/2021.acl-long.203
DOI:
10.18653/v1/2021.acl-long.203
Bibkey:
Cite (ACL):
Jinming Zhao, Ruichen Li, and Qin Jin. 2021. Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2608–2618, Online. Association for Computational Linguistics.
Cite (Informal):
Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities (Zhao et al., ACL-IJCNLP 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.acl-long.203.pdf
Video:
 https://aclanthology.org/2021.acl-long.203.mp4
Code
 aim3-ruc/mmin
Data
IEMOCAPMSP-IMPROV