Are Gestures Worth a Thousand Words? An Analysis of Interviews in the Political Domain

Daniela Trotta, Sara Tonelli


Abstract
Speaker gestures are semantically co-expressive with speech and serve different pragmatic functions to accompany oral modality. Therefore, gestures are an inseparable part of the language system: they may add clarity to discourse, can be employed to facilitate lexical retrieval and retain a turn in conversations, assist in verbalizing semantic content and facilitate speakers in coming up with the words they intend to say. This aspect is particularly relevant in political discourse, where speakers try to apply communication strategies that are both clear and persuasive using verbal and non-verbal cues. In this paper we investigate the co-speech gestures of several Italian politicians during face-to-face interviews using a multimodal linguistic approach. We first enrich an existing corpus with a novel annotation layer capturing the function of hand movements. Then, we perform an analysis of the corpus, focusing in particular on the relationship between hand movements and other information layers such as the political party or non-lexical and semi-lexical tags. We observe that the recorded differences pertain more to single politicians than to the party they belong to, and that hand movements tend to occur frequently with semi-lexical phenomena, supporting the lexical retrieval hypothesis.
Anthology ID:
2021.mmsr-1.2
Volume:
Proceedings of the 1st Workshop on Multimodal Semantic Representations (MMSR)
Month:
June
Year:
2021
Address:
Groningen, Netherlands (Online)
Editors:
Lucia Donatelli, Nikhil Krishnaswamy, Kenneth Lai, James Pustejovsky
Venue:
MMSR
SIG:
SIGSEM
Publisher:
Association for Computational Linguistics
Note:
Pages:
11–20
Language:
URL:
https://aclanthology.org/2021.mmsr-1.2
DOI:
Bibkey:
Cite (ACL):
Daniela Trotta and Sara Tonelli. 2021. Are Gestures Worth a Thousand Words? An Analysis of Interviews in the Political Domain. In Proceedings of the 1st Workshop on Multimodal Semantic Representations (MMSR), pages 11–20, Groningen, Netherlands (Online). Association for Computational Linguistics.
Cite (Informal):
Are Gestures Worth a Thousand Words? An Analysis of Interviews in the Political Domain (Trotta & Tonelli, MMSR 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.mmsr-1.2.pdf