How Much Does Prosody Help Turn-taking? Investigations using Voice Activity Projection Models

Erik Ekstedt, Gabriel Skantze


Abstract
Turn-taking is a fundamental aspect of human communication and can be described as the ability to take turns, project upcoming turn shifts, and supply backchannels at appropriate locations throughout a conversation. In this work, we investigate the role of prosody in turn-taking using the recently proposed Voice Activity Projection model, which incrementally models the upcoming speech activity of the interlocutors in a self-supervised manner, without relying on explicit annotation of turn-taking events, or the explicit modeling of prosodic features. Through manipulation of the speech signal, we investigate how these models implicitly utilize prosodic information. We show that these systems learn to utilize various prosodic aspects of speech both on aggregate quantitative metrics of long-form conversations and on single utterances specifically designed to depend on prosody.
Anthology ID:
2022.sigdial-1.51
Volume:
Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue
Month:
September
Year:
2022
Address:
Edinburgh, UK
Editors:
Oliver Lemon, Dilek Hakkani-Tur, Junyi Jessy Li, Arash Ashrafzadeh, Daniel Hernández Garcia, Malihe Alikhani, David Vandyke, Ondřej Dušek
Venue:
SIGDIAL
SIG:
SIGDIAL
Publisher:
Association for Computational Linguistics
Note:
Pages:
541–551
Language:
URL:
https://aclanthology.org/2022.sigdial-1.51
DOI:
10.18653/v1/2022.sigdial-1.51
Bibkey:
Cite (ACL):
Erik Ekstedt and Gabriel Skantze. 2022. How Much Does Prosody Help Turn-taking? Investigations using Voice Activity Projection Models. In Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 541–551, Edinburgh, UK. Association for Computational Linguistics.
Cite (Informal):
How Much Does Prosody Help Turn-taking? Investigations using Voice Activity Projection Models (Ekstedt & Skantze, SIGDIAL 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.sigdial-1.51.pdf
Video:
 https://youtu.be/QstNOaBfJ5k
Code
 erikekstedt/conv_ssl +  additional community code