DialogueTRM: Exploring Multi-Modal Emotional Dynamics in a Conversation

Yuzhao Mao, Guang Liu, Xiaojie Wang, Weiguo Gao, Xuan Li


Abstract
Emotion dynamics formulates principles explaining the emotional fluctuation during conversations. Recent studies explore the emotion dynamics from the self and inter-personal dependencies, however, ignoring the temporal and spatial dependencies in the situation of multi-modal conversations. To address the issue, we extend the concept of emotion dynamics to multi-modal settings and propose a Dialogue Transformer for simultaneously modeling the intra-modal and inter-modal emotion dynamics. Specifically, the intra-modal emotion dynamics is to not only capture the temporal dependency but also satisfy the context preference in every single modality. The inter-modal emotional dynamics aims at handling multi-grained spatial dependency across all modalities. Our models outperform the state-of-the-art with a margin of 4%-16% for most of the metrics on three benchmark datasets.
Anthology ID:
2021.findings-emnlp.229
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2021
Month:
November
Year:
2021
Address:
Punta Cana, Dominican Republic
Editors:
Marie-Francine Moens, Xuanjing Huang, Lucia Specia, Scott Wen-tau Yih
Venue:
Findings
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
2694–2704
Language:
URL:
https://aclanthology.org/2021.findings-emnlp.229
DOI:
10.18653/v1/2021.findings-emnlp.229
Bibkey:
Cite (ACL):
Yuzhao Mao, Guang Liu, Xiaojie Wang, Weiguo Gao, and Xuan Li. 2021. DialogueTRM: Exploring Multi-Modal Emotional Dynamics in a Conversation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2694–2704, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Cite (Informal):
DialogueTRM: Exploring Multi-Modal Emotional Dynamics in a Conversation (Mao et al., Findings 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.findings-emnlp.229.pdf
Video:
 https://aclanthology.org/2021.findings-emnlp.229.mp4
Data
IEMOCAPMELD