Beyond Fresh Starts: Stateful Inference for Streaming ASR in Conversational Voice Agents

Sameep Chattopadhyay, Alexander Erdmann, Mari Ostendorf


Abstract
Modern voice-agent systems rely on streaming speech recognition models that operate under stringent latency constraints. This study shows that, due to the limited memory constraints of real-time processing, these systems are adversely impacted by conversational phenomena such as long silences and backchannels. While many agentic pipelines mitigate this by resetting state at each turn, this approach discards vital context and impairs performance at turn onsets. We propose two state-management strategies that preserve cross-utterance context to reduce onset errors. In experiments with two state-of-the-art streaming models on two spoken dialogue benchmarks, our best method yields an average of 15-21% relative WER reduction at utterance onsets.
Anthology ID:
2026.emnlp-main.1703
Volume:
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Month:
October
Year:
2026
Address:
Budapest, Hungary
Editors:
Hung-Yi Lee, Sunipa Dev, Jessy Li, Desmond Elliott
Venue:
EMNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
38802–38811
Language:
URL:
https://aclanthology.org/2026.emnlp-main.1703/
DOI:
10.18653/2026.emnlp-main.1703
Bibkey:
Cite (ACL):
Sameep Chattopadhyay, Alexander Erdmann, and Mari Ostendorf. 2026. Beyond Fresh Starts: Stateful Inference for Streaming ASR in Conversational Voice Agents. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 38802–38811, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):
Beyond Fresh Starts: Stateful Inference for Streaming ASR in Conversational Voice Agents (Chattopadhyay et al., EMNLP 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.emnlp-main.1703.pdf