<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="berner-etal-2026-exploring">
    <titleInfo>
        <title>Exploring Multimodal Turn-Taking Cues in Face-to-Face Conversation using Voice Activity Projection</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Willem</namePart>
        <namePart type="family">Berner</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Julio</namePart>
        <namePart type="given">Cesar</namePart>
        <namePart type="family">Cavalcanti</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Kalle</namePart>
        <namePart type="family">Åström</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Gabriel</namePart>
        <namePart type="family">Skantze</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-10</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 1st Workshop on Multimodal Interaction in Face-to-Face Dialogue</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Esam</namePart>
            <namePart type="family">Ghaleb</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Ece</namePart>
            <namePart type="family">Takmaz</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Diego</namePart>
            <namePart type="family">Frassinelli</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Bulat</namePart>
            <namePart type="family">Khaertdinov</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Raquel</namePart>
            <namePart type="family">Fernández</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Zerrin</namePart>
            <namePart type="family">Yumak</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Asli</namePart>
            <namePart type="family">Ozyurek</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Budapest, Hungary</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-482-8</identifier>
    </relatedItem>
    <abstract>Turn-taking is a fundamental component of spoken interaction, and while humans naturally rely on both verbal and non-verbal signals, dialogue systems usually depend on audio cues alone. This paper investigates whether visual features from face-to-face conversations can enhance turn-taking prediction beyond what is achievable from audio-only. We extend the Voice Activity Projection (VAP) model, a self-supervised transformer-based model for predicting future voice activity, by incorporating visual features extracted from the large-scale Meta Seamless Interaction dataset of dyadic face-to-face conversations. The visual features include gaze direction, head movement, body and hand pose, and facial action units (FAU). For incorporating the visual features, we explore concatenation, cross-attention fusion, delta features, and trainable gating mechanisms. Results show that visual information improves performance over the audio-only baseline, with FAU being significantly more informative than other feature groups. Body and gaze features nevertheless contribute complementary information, as the model combining all features performs best. Furthermore, results indicate that performance on specific tasks varies depending on whether training and test data come from improvised (acted) or naturalistic (non-acted) conversations.</abstract>
    <identifier type="citekey">berner-etal-2026-exploring</identifier>
    <location>
        <url>https://aclanthology.org/2026.mint-1.6/</url>
    </location>
    <part>
        <date>2026-10</date>
        <extent unit="page">
            <start>50</start>
            <end>58</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Exploring Multimodal Turn-Taking Cues in Face-to-Face Conversation using Voice Activity Projection
%A Berner, Willem
%A Cavalcanti, Julio Cesar
%A Åström, Kalle
%A Skantze, Gabriel
%Y Ghaleb, Esam
%Y Takmaz, Ece
%Y Frassinelli, Diego
%Y Khaertdinov, Bulat
%Y Fernández, Raquel
%Y Yumak, Zerrin
%Y Ozyurek, Asli
%S Proceedings of the 1st Workshop on Multimodal Interaction in Face-to-Face Dialogue
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-482-8
%F berner-etal-2026-exploring
%X Turn-taking is a fundamental component of spoken interaction, and while humans naturally rely on both verbal and non-verbal signals, dialogue systems usually depend on audio cues alone. This paper investigates whether visual features from face-to-face conversations can enhance turn-taking prediction beyond what is achievable from audio-only. We extend the Voice Activity Projection (VAP) model, a self-supervised transformer-based model for predicting future voice activity, by incorporating visual features extracted from the large-scale Meta Seamless Interaction dataset of dyadic face-to-face conversations. The visual features include gaze direction, head movement, body and hand pose, and facial action units (FAU). For incorporating the visual features, we explore concatenation, cross-attention fusion, delta features, and trainable gating mechanisms. Results show that visual information improves performance over the audio-only baseline, with FAU being significantly more informative than other feature groups. Body and gaze features nevertheless contribute complementary information, as the model combining all features performs best. Furthermore, results indicate that performance on specific tasks varies depending on whether training and test data come from improvised (acted) or naturalistic (non-acted) conversations.
%U https://aclanthology.org/2026.mint-1.6/
%P 50-58

Markdown (Informal)

[Exploring Multimodal Turn-Taking Cues in Face-to-Face Conversation using Voice Activity Projection](https://aclanthology.org/2026.mint-1.6/) (Berner et al., MINT 2026)

• Exploring Multimodal Turn-Taking Cues in Face-to-Face Conversation using Voice Activity Projection (Berner et al., MINT 2026)

ACL

• Willem Berner, Julio Cesar Cavalcanti, Kalle Åström, and Gabriel Skantze. 2026. Exploring Multimodal Turn-Taking Cues in Face-to-Face Conversation using Voice Activity Projection. In Proceedings of the 1st Workshop on Multimodal Interaction in Face-to-Face Dialogue, pages 50–58, Budapest, Hungary. Association for Computational Linguistics.