Prathusha K Sarma


2021

pdf bib
A New View of Multi-modal Language Analysis: Audio and Video Features as Text “Styles”
Zhongkai Sun | Prathusha K Sarma | Yingyu Liang | William Sethares
Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume

Imposing the style of one image onto another is called style transfer. For example, the style of a Van Gogh painting might be imposed on a photograph to yield an interesting hybrid. This paper applies the adaptive normalization used for image style transfer to language semantics, i.e., the style is the way the words are said (tone of voice and facial expressions) and these are style-transferred onto the text. The goal is to learn richer representations for multi-modal utterances using style-transferred multi-modal features. The proposed Style-Transfer Transformer (STT) grafts a stepped styled adaptive layer-normalization onto a transformer network, the output from which is used in sentiment analysis and emotion recognition problems. In addition to achieving performance on par with the state-of-the art (but using less than a third of the model parameters), we examine the relative contributions of each mode when used in the downstream applications.