State-of-the-art dialogue systems focus on two-person dialogue and explicit indication of who is speaking. However, for using LLM-based AI systems in multi-speaker environments, it is necessary to determine who is speaking next. In this work, we investigate how the task can be formulated and how multi-modal LLMs are able to perform it. First, we frame the task as multi-modal Next Speaker Prediction (mNSP) and create a multi-modal, multi-speaker NSP dataset from multiple existing dialogue datasets with different task setups. Second, we build dedicated systems for the mNSP task using multi-modal LLMs, and finally perform automatic evaluation, comparing them to rule-based baselines.

Real-time communication mediated by artificial intelligence (AI) can produce an accurate output while changing how a speaker plans an utterance and what a listener receives. Output measures, such as word error rate, machine translation scores, and task accuracy, leave these process effects unobserved. We propose a dual-axis framework that aligns disfluency measures with suggestion correspondence, including lexical entrainment and semantic similarity. Disfluency type, rate, timing, and duration characterize observable production difficulty. Correspondence measures characterize whether suggestion content appears in the response. Observing both axes within the same episode distinguishes behavioral configurations that either family alone leaves ambiguous and supports explicit hypotheses about planning, uptake, and resistance. An illustrative corpus demonstrates event-detection feasibility and shows that semantic correspondence varies with coded incorporation and exceeds a comparison matched on presentation format. These checks leave cognitive interpretations open to experimental validation. We also derive a prediction concerning listeners: removing natural disfluencies may remove cues that listeners use to anticipate difficult or novel material. The resulting framework and reporting protocol extend standard evaluation to the human processes through which real-time AI-mediated communication succeeds or fails.

Child–robot interaction places real demands on perception systems: a robot engaged in play with a child cannot afford to miss the child’s focus of attention, yet running every perception module at every frame is computationally expensive on the kind of edge hardware that privacy-respecting robots must use. This paper addresses this tension by proposing a cadence-stratified multimodal perception pipeline, WordMate-CRI, in which each module runs at a rate that trades computation against staleness, informed by how long each cue remains stable for a potential interaction partner. Detection and tracking run continuously; action recognition uses rolling temporal buffers; emotion is refreshed periodically; and attributes such as age and gender are estimated once at the start of a session and cached. We evaluated WordMate-CRI through an LLM response-quality ablation study across six toy-play scenarios and a latency benchmark on Jetson AGX Orin hardware. The ablation shows that VLM enrichment and object evidence are the dominant grounding signals, with WordMate-CRI achieving a mean scene-grounded score of 0.967. On live Jetson input, cadence-based scheduling raises throughput from 4.3 to 18.2 frames per second (FPS), clearing the 15 FPS deployment target, while reducing WordMate-CRI latency from 6411.5 to 5844.5 ms. Taken together, the results make a practical case that deciding when each perception module should run matters just as much as deciding what it should recognise. This optimisation is important for systems aiming to produce contingent and timely child-directed speech in multi-modal contexts.

Humor recognition in conversations is a highly challenging task due to its reliance on context, sarcasm, timing, and multimodal cues. While several humour datasets exist for English, resources for Indic languages like Hindi remain little scarce, limiting the development of culturally aware conversational AI systems. To address this gap, we introduce a new multimodal dataset for humour detection from conversational dialogues. The dataset consists of 5203 annotated utterances of 2 season containing 8 episodes each from a popular Indian web series Panchayat. Utterances are kept in their original Devanagari script as spoken in the series, without transliteration or standardization into pure Hindi, to retain the natural conversational tone and cultural nuances. Each utterance is annotated with a binary humour label (Humour / Not Humour) and includes aligned textual, acoustic and visual information, making it suitable for multimodal and context-aware humour research. Our experiments highlight the complementary nature of multimodal signals and the importance of conversational context in understanding Hindi humour. This dataset is expected to promote research on computational humour, sarcasm detection and multimodal dialogue understanding for low-resource languages. The dataset and baselines are available at https://huggingface.co/datasets/Abhis4e/Panchayat-multimodal-dataset and https://github.com/singhbishtabhishek/Multimodal-Dataset.

Whispering differs from voiced speech in the lack of fundamental frequency (F0) and is characterized by substantially reduced amplitude. This production study investigates how head movements are modulated in whispering when speakers are visible or not, while producing polar questions with rising intonation versus statements with flat intonation. An experiment with 15 German speakers shows that head movements are more pronounced in questions than in statements, with the strongest effects observed in normally spoken questions under mutual visibility. Overall, the findings suggest head movements serve as a flexible multimodal prosodic resource, aligning with acoustic cues in normal speech under favorable communicative conditions while functioning more independently as compensatory mechanism in whispered speech.

Turn-taking is a fundamental component of spoken interaction, and while humans naturally rely on both verbal and non-verbal signals, dialogue systems usually depend on audio cues alone. This paper investigates whether visual features from face-to-face conversations can enhance turn-taking prediction beyond what is achievable from audio-only. We extend the Voice Activity Projection (VAP) model, a self-supervised transformer-based model for predicting future voice activity, by incorporating visual features extracted from the large-scale Meta Seamless Interaction dataset of dyadic face-to-face conversations. The visual features include gaze direction, head movement, body and hand pose, and facial action units (FAU). For incorporating the visual features, we explore concatenation, cross-attention fusion, delta features, and trainable gating mechanisms. Results show that visual information improves performance over the audio-only baseline, with FAU being significantly more informative than other feature groups. Body and gaze features nevertheless contribute complementary information, as the model combining all features performs best. Furthermore, results indicate that performance on specific tasks varies depending on whether training and test data come from improvised (acted) or naturalistic (non-acted) conversations.

Large-scale multimodal communication research increasingly depends on computational methods capable of extracting reliable gesture annotations from video data. Yet it remains unclear to what extent automatically generated annotations can support the linguistic investigation of naturally occurring co-speech gesture and where expert interpretation remains indispensable. This paper presents a framework for the automatic analysis and annotation of co-speech gesture in media corpora, developed to support multimodal communication research at scale. We build our framework on top of the existing computer vision frameworks for keypoint extraction and active speaker detection, with post-processing steps carefully designed to mitigate noisy output. Building on the processed keypoint trajectories, we implement a configurable rule-based annotation system developed through close collaboration between computer vision researchers and linguists. We demonstrate the usefulness of the framework through a case study investigating vertical hand movement accompanying English expressions of future and past, examining which aspects of this gestural pattern can be captured through automatic annotation and which require expert multimodal interpretation. We find that at corpus scale, greater mean amplitude and higher mean peak velocity converge in *will*-future, suggesting greater kinematic prominence in the multimodal expression of futurity. More broadly, the case study demonstrates how computational pipelines can support, rather than replace, human analysis by enabling multimodal hypotheses to be investigated on datasets that would be impractical to analyse manually.

Fréchet Gesture Distance (FGD) is the standard automatic metric for co-speech gesture generation, the models that synthesize the hand and body motion accompanying speech. Human perceptual studies remain the gold standard, and the GENEA challenges rank systems by them, but such studies are slow and costly, so papers routinely compare systems by FGD instead. FGD is a Fréchet distance in the latent space of a learned pose autoencoder, so it inherits the many choices behind that encoder, most of which papers never report. We ask whether FGD is a reliable proxy for perceptual evaluation under these unreported choices. Holding the systems fixed, we vary the reference tool’s own settings (analysis window, seed, feature versus raw space) and the corpus the feature extractor is trained on, on both tiers of the GENEA 2022 evaluation and on the GENEA 2023 challenge. First, the FGD ranking is not reproducible where it matters: the "best" system is one of two that a per-clip bootstrap shows are statistically tied, so which one wins is a configuration choice, though aggregate rank correlation stays moderate rather than collapsing. Second, and more consequential, FGD’s agreement with human ratings is modest and itself tunable, moving by 0.18 to 0.24 Kendall τ across these undocumented choices: FGD can be made to look appreciably more or less human-valid without changing the systems. These effects replicate across two challenge years and three evaluation tiers. We conclude that FGD numbers are not comparable across papers unless the window, seed, scoring space, and feature-extractor training corpus are pinned and reported, and that FGD cannot resolve small differences between closely performing systems.

Engagement detection in meetings has focused mostly on speakers, who produce explicit prosodic and acoustic cues. The engagement of non-speaking participants is harder to read because it surfaces only in nonverbal behaviour. We predict how engaged a silent listener is, on a three-level scale (Engaged, Intermittent, Disengaged), in five-second windows of remote Zoom meetings from the GEstures and Head Movements in Language (GEHM) corpus. Each window is described by three feature families: the listener’s own nonverbal behaviour, the listener’s behavioural alignment with the others present, including the active speaker, and that speaker’s prosody. Interpersonal coordination is an established correlate of rapport and involvement. We ask the narrower question of whether a compact window-level alignment measure adds predictive value over listener-only features when confident labels are scarce. Under leakage-free cross-validation, listener-speaker alignment is the most useful addition we test, and the speaker’s terminal pitch slope is an interpretable correlate of listener engagement. The gains are modest and their bootstrap intervals include zero. Unsupervised analysis finds no engagement structure. Two findings are firmer: the task is effectively binary, actively engaged versus not, bounded by the scarcity of confident disengaged labels, and the pitch-slope correlate holds despite the small sample.

Language technology resources are limited for signed languages. In order to improve these technologies, large amounts of labeled language data stored in annotated corpora are necessary. Many existing annotated corpora in American Sign Language (ASL) do not include non-manual markers (NMMs), i.e., language features that do not involve the hands. This work underscores the importance of including NMMs in sign language research by describing their plurifunctional nature in signed conversation, specifying their link to sign language prosody, examining the current state of ASL corpus research, and recommending a systematized method of annotation which captures the critical but complex variations in NMM production in ASL.

Discrete motion tokenizers encode motion as atomic units and are widely used for co-speech gesture generation. It remains unclear which motion properties, especially those relevant to gesture semantics, are recoverable from these codebooks. We probe a reconstruction-trained codebook using 19 co-speech gesture descriptors spanning from raw motion to abstract communicative function. Results show that geometry and handedness are readily decodable from token embeddings, while motion category is only weakly decoded despite showing systematic differences in discrete code usage. This gap between reconstruction quality and descriptor decodability suggests that reconstruction objectives alone do not guarantee that gesture semantics are captured, and that evaluating codebooks on such properties can guide the design of more semantic motion tokenizers.

Robots that search for objects on their own must decide what to do when the search fails, and how they signal that failure to a human partner can determine whether the interaction turns into collaboration. Prior work on human–robot interaction has studied nonverbal help-seeking and social gaze separately, but rarely compares gaze against a matched verbal failure signal inside a live, real-time embodied search task. We present an embodied dialogue system, built on the Retico incremental processing framework, where the Cozmo robot searches for a specified object and signals failure either verbally or through gaze towards the human partner. We first compare a rule-based planner against a vision-language model (VLM) planner that reasons over scene semantics, finding that the VLM planner succeeds far more often (87.2% vs. 65.9%). We then test whether gaze can close this gap without adding reasoning capability to the rule-based planner. Adding gaze as the stuck signal raises success to 87.87%, matching the VLM planner, while a verbal stuck signal only raises it slightly (68.75%). Results further show that this effect is largely implicit: participants helped more in the gaze condition without consistently reporting that they felt invited to do so. Our findings suggest that gaze can substitute for reasoning capability a robot lacks, helping it elicit assistance from a human partner.

Storytelling in face-to-face dialogue involves coordinating multiple modalities, including speech and gaze. When the addressee is a social robot rather than a human, narrators’ speech and gaze behavior may differ. However, few studies have examined both modalities within the same narratives, and it remains unclear how different aspects of multimodal behavior vary depending on whether the interaction partner is a social robot or a human. In this study, we examined how adults narrated picture-based stories to a social robot and a human interlocutor. Two modalities of narrative production were considered: pausing, separated into filled and unfilled pauses, and gaze behavior during the narration, directed either toward the story pictures or toward the listener. Critically, we examined whether shyness, considered as a continuous dimension of individual variation, relates to pausing and gaze behavior. Results revealed that speakers produced fewer filled pauses when narrating to the robot than to the human, while unfilled pauses did not differ. Additionally, higher shyness was associated with longer, though not more frequent, filled pauses in human-directed narration, although shyness did not reliably predict the magnitude of the partner effect. We discuss how our findings may contribute to a more nuanced understanding of multimodal interactions with embodied agents.

Reading a social situation often depends on behavior, not words alone. We introduce FriendBench, a benchmark for inferring whether two people are already familiar or are meeting as strangers, from a 20-second clip of a dyadic ice-breaker conversation. Every pair answers the same type of prompt, so only the manner of interaction can reveal the answer. Across text, audio, and video, we compare 26 models from seven companies against matched human panels over 96 balanced dyads. The best model and the human crowd are statistically indistinguishable on accuracy in every modality, but reach it differently: humans stay balanced across the two answers, while the strongest models favor "stranger." This is a difference in effective prior, not in discrimination. Richer channels help both unequally, and only humans gain from visible behavior on top of speech. We release the stimuli, human ratings, and model predictions.

Mental health, defined by psychological, emotional, and social well-being, has emerged as a significant social issue in modern society. To assess an individual’s mental state and provide personalized support, clinician–patient interviews are commonly conducted. However, given the constraints on access to such interviews, conversational agents have the potential to provide more accessible mental health support. In this paper, we present , a multimodal conversational agent that assesses an individual’s emotional and depressive states using an audio–text model that jointly models linguistic and paralinguistic cues from speech, and provides personalized mental health support through LLM-based response generation. Quantitative and qualitative evaluations indicate improvements in multimodal state assessment and response quality over the considered baselines.

Recurrent gestures are conventionalized gestures with a stable relationship between form and meaning. One such gesture is the palm up open hand (PUOH) gesture, typically produced by extending the fingers and rotating the lower arm so that the palm faces upward. The gesture is attested across many languages and cultures and is often used to present a point, yet little is known about the forms and functions it takes in Dutch. Automatic gesture recognition could reduce the manual labor that quantitative gesture studies require, but recognizing this gesture is difficult, partly because of the large amount of variation present in its form. This paper compares two gesture recognition approaches on a newly collected corpus of naturalistic Dutch dyadic conversation: a static model that takes single images as input and a dynamic model that takes video clips as input. Both models are trained and tested with varying amounts of gesture-form variation included in the data. The static model is found to be less affected by this variation and to yield overall better results than the dynamic model. Our results illustrate some of the central challenges that gesture recognition systems face in recognizing the PUOH gesture and offer insights that may improve performance in future studies of multimodal, face-to-face behavior.

Personality is a key factor in human–agent and human-robot interaction, influencing user perception, trust, and engagement. This work investigates methods for inducing Big Five personality traits in (open) language models, with a focus on smaller variants as more easily deployable in real-time agents and robotic systems. We compare prompt-based conditioning, parameter-efficient fine-tuning, and activation steering, focusing on their behaviour in multi-turn dialogue and embodied gesture generation. In particular, we investigate personality drift across turns and non-verbal behaviour generation through direct function calls.

We find that classifier-assigned trait alignment differs little across methods, with no consistent evidence of trait drift over multi-turn interaction. However, conditioning method significantly affects dialogue quality and robustness, with steering and fine-tuning introducing artefacts and instability. Interestingly, the more consequential differences emerge in the multimodal channel. Conditioning method, not intended personality, is the dominant driver of gesture behaviour, reshaping gesture diversity, tool-use reliability, and output-gesture congruence. Our findings suggest that personality conditioning should be evaluated jointly across verbal and non-verbal channels, and that method choice is itself a design decision shaping the expressiveness and reliability of embodied conversational agents.

In collaborative tasks with asymmetric information, participants coordinate their understanding through interaction. We ask whether gaze provides evidence about grounding across two such tasks. Working from discrete behavioral annotations, we map HCRC MapTask (Anderson et al., 1991) and MUNDEX (Türk et al., 2023) into a shared partner/task/away vocabulary and compute gaze features around task-relevant dialogue units. In both corpora, aligned reference interpretations (MapTask) and UND (understood) judgments (MUNDEX) are associated with more task-directed gaze and with less partner-directed gaze, lower gaze entropy, and fewer gaze transitions. The associations are clearest for the participant leading the task: in giver-produced references, and in explainer judgments, which also co-vary with the explainee’s gaze. In same-speaker MapTask reference chains, the speaker’s gaze entropy is lower at the mention where a previously non-aligned referent becomes aligned. The best gaze feature groups improve modestly over controls under grouped cross-validation: temporal features in MapTask and raw proportions in MUNDEX. Because effects are small and several weaken when recurring participants rather than dialogues are the unit of inference, we treat gaze as one contributing cue to grounding, to be interpreted alongside task and dialogue context.