From Output Metrics to Cognitive Process: Disfluency and Lexical Entrainment for Evaluating Real-Time AI-Mediated Communication

Wanqing Psyche He, Susan Fussell


Abstract
Real-time communication mediated by artificial intelligence (AI) can produce an accurate output while changing how a speaker plans an utterance and what a listener receives. Output measures, such as word error rate, machine translation scores, and task accuracy, leave these process effects unobserved. We propose a dual-axis framework that aligns disfluency measures with suggestion correspondence, including lexical entrainment and semantic similarity. Disfluency type, rate, timing, and duration characterize observable production difficulty. Correspondence measures characterize whether suggestion content appears in the response. Observing both axes within the same episode distinguishes behavioral configurations that either family alone leaves ambiguous and supports explicit hypotheses about planning, uptake, and resistance. An illustrative corpus demonstrates event-detection feasibility and shows that semantic correspondence varies with coded incorporation and exceeds a comparison matched on presentation format. These checks leave cognitive interpretations open to experimental validation. We also derive a prediction concerning listeners: removing natural disfluencies may remove cues that listeners use to anticipate difficult or novel material. The resulting framework and reporting protocol extend standard evaluation to the human processes through which real-time AI-mediated communication succeeds or fails.
Anthology ID:
2026.mint-1.2
Volume:
Proceedings of the 1st Workshop on Multimodal Interaction in Face-to-Face Dialogue
Month:
October
Year:
2026
Address:
Budapest, Hungary
Editors:
Esam Ghaleb, Ece Takmaz, Diego Frassinelli, Bulat Khaertdinov, Raquel Fernández, Zerrin Yumak, Asli Ozyurek
Venues:
MINT | WS
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
10–21
Language:
URL:
https://aclanthology.org/2026.mint-1.2/
DOI:
Bibkey:
Cite (ACL):
Wanqing Psyche He and Susan Fussell. 2026. From Output Metrics to Cognitive Process: Disfluency and Lexical Entrainment for Evaluating Real-Time AI-Mediated Communication. In Proceedings of the 1st Workshop on Multimodal Interaction in Face-to-Face Dialogue, pages 10–21, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):
From Output Metrics to Cognitive Process: Disfluency and Lexical Entrainment for Evaluating Real-Time AI-Mediated Communication (He & Fussell, MINT 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.mint-1.2.pdf