Multimodal Routing: Improving Local and Global Interpretability of Multimodal Language Analysis

Yao-Hung Hubert Tsai, Martin Ma, Muqiao Yang, Ruslan Salakhutdinov, Louis-Philippe Morency


Abstract
The human language can be expressed through multiple sources of information known as modalities, including tones of voice, facial gestures, and spoken language. Recent multimodal learning with strong performances on human-centric tasks such as sentiment analysis and emotion recognition are often black-box, with very limited interpretability. In this paper we propose, which dynamically adjusts weights between input modalities and output representations differently for each input sample. Multimodal routing can identify relative importance of both individual modalities and cross-modality factors. Moreover, the weight assignment by routing allows us to interpret modality-prediction relationships not only globally (i.e. general trends over the whole dataset), but also locally for each single input sample, meanwhile keeping competitive performance compared to state-of-the-art methods.
Anthology ID:
2020.emnlp-main.143
Volume:
Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
Month:
November
Year:
2020
Address:
Online
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1823–1833
Language:
URL:
https://aclanthology.org/2020.emnlp-main.143
DOI:
10.18653/v1/2020.emnlp-main.143
Bibkey:
Copy Citation:
PDF:
https://aclanthology.org/2020.emnlp-main.143.pdf
Optional supplementary material:
 2020.emnlp-main.143.OptionalSupplementaryMaterial.zip
Video:
 https://slideslive.com/38938697
Code
 martinmamql/multimodal_routing
Data
CMU-MOSEIIEMOCAP