Gestures Are Used Rationally: Information Theoretic Evidence from Neural Sequential Models

Yang Xu, Yang Cheng, Riya Bhatia


Abstract
Verbal communication is companied by rich non-verbal signals. The usage of gestures, poses, and facial expressions facilitates the information transmission in verbal channel. However, few computational studies have explored the non-verbal channels with finer theoretical lens. We extract gesture representations from monologue video data and train neural sequential models, in order to study the degree to which non-verbal signals can effectively transmit information. We focus on examining whether the gestures demonstrate the similar pattern of entropy rate constancy (ERC) found in words, as predicted by Information Theory. Positive results are shown to support the assumption, which leads to the conclusion that speakers indeed use simple gestures to convey information that enhances verbal communication, and the production of non-verbal information is rationally organized.
Anthology ID:
2022.coling-1.12
Volume:
Proceedings of the 29th International Conference on Computational Linguistics
Month:
October
Year:
2022
Address:
Gyeongju, Republic of Korea
Venue:
COLING
SIG:
Publisher:
International Committee on Computational Linguistics
Note:
Pages:
134–140
Language:
URL:
https://aclanthology.org/2022.coling-1.12
DOI:
Bibkey:
Cite (ACL):
Yang Xu, Yang Cheng, and Riya Bhatia. 2022. Gestures Are Used Rationally: Information Theoretic Evidence from Neural Sequential Models. In Proceedings of the 29th International Conference on Computational Linguistics, pages 134–140, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Cite (Informal):
Gestures Are Used Rationally: Information Theoretic Evidence from Neural Sequential Models (Xu et al., COLING 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.coling-1.12.pdf