A New View of Multi-modal Language Analysis: Audio and Video Features as Text “Styles”

Zhongkai Sun, Prathusha K Sarma, Yingyu Liang, William Sethares


Abstract
Imposing the style of one image onto another is called style transfer. For example, the style of a Van Gogh painting might be imposed on a photograph to yield an interesting hybrid. This paper applies the adaptive normalization used for image style transfer to language semantics, i.e., the style is the way the words are said (tone of voice and facial expressions) and these are style-transferred onto the text. The goal is to learn richer representations for multi-modal utterances using style-transferred multi-modal features. The proposed Style-Transfer Transformer (STT) grafts a stepped styled adaptive layer-normalization onto a transformer network, the output from which is used in sentiment analysis and emotion recognition problems. In addition to achieving performance on par with the state-of-the art (but using less than a third of the model parameters), we examine the relative contributions of each mode when used in the downstream applications.
Anthology ID:
2021.eacl-main.167
Volume:
Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume
Month:
April
Year:
2021
Address:
Online
Editors:
Paola Merlo, Jorg Tiedemann, Reut Tsarfaty
Venue:
EACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1956–1965
Language:
URL:
https://aclanthology.org/2021.eacl-main.167
DOI:
10.18653/v1/2021.eacl-main.167
Bibkey:
Cite (ACL):
Zhongkai Sun, Prathusha K Sarma, Yingyu Liang, and William Sethares. 2021. A New View of Multi-modal Language Analysis: Audio and Video Features as Text “Styles”. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1956–1965, Online. Association for Computational Linguistics.
Cite (Informal):
A New View of Multi-modal Language Analysis: Audio and Video Features as Text “Styles” (Sun et al., EACL 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.eacl-main.167.pdf
Data
CMU-MOSEIIEMOCAP