Sameep Chattopadhyay
Author directory2026
Beyond Fresh Starts: Stateful Inference for Streaming ASR in Conversational Voice Agents
Sameep Chattopadhyay | Alexander Erdmann | Mari Ostendorf
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Sameep Chattopadhyay | Alexander Erdmann | Mari Ostendorf
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Modern voice-agent systems rely on streaming speech recognition models that operate under stringent latency constraints. This study shows that, due to the limited memory constraints of real-time processing, these systems are adversely impacted by conversational phenomena such as long silences and backchannels. While many agentic pipelines mitigate this by resetting state at each turn, this approach discards vital context and impairs performance at turn onsets. We propose two state-management strategies that preserve cross-utterance context to reduce onset errors. In experiments with two state-of-the-art streaming models on two spoken dialogue benchmarks, our best method yields an average of 15-21% relative WER reduction at utterance onsets.
The Cost of Sounding Different: Accent Bias in Audio Language Models
Aruna Srivastava | Orevaoghene Ahia | Li Lucy | Ashley Christendat | Sameep Chattopadhyay | Farhan Samir | Tejumade Afonja | Valentin Hofmann | Sachin Kumar | Noah A. Smith | Yulia Tsvetkov
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Aruna Srivastava | Orevaoghene Ahia | Li Lucy | Ashley Christendat | Sameep Chattopadhyay | Farhan Samir | Tejumade Afonja | Valentin Hofmann | Sachin Kumar | Noah A. Smith | Yulia Tsvetkov
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Audio language models (LMs) are now integrated into high-stakes, decision-oriented settings: hiring pipelines, oral assessments, and language proficiency tests. Prior work in natural language processing has primarily focused on how LMs respond to socially-driven textual variation. With the growing use of LMs that natively process audio, similar robustness and fairness issues arise for speech variation. In our work, we examine how accent variation in English inputs affects audio LMs’ assessments of speakers. We evaluate six audio LMs on speech across five accent groups: American, British, Chinese, Indian, and Nigerian English. We find that several LMs assign lower speech delivery scores to Chinese- and Nigerian-accented speakers, regardless of content quality. At the individual level, delivery scores correlate negatively with acoustic distance from American English across three widely-used models (Spearman ρ = −0.35 to −0.66, all p < .05), suggesting that English speech is scored against an implicit American acoustic norm. Even though models are able to transcribe the speech in our studies, their assessment and perceptions of speakers may result in allocational harm. We release our framework and corpus to support future analyses of bias in audio LMs.
2025
AMPS: ASR with Multimodal Paraphrase Supervision
Abhishek Gupta | Amruta Parulekar | Sameep Chattopadhyay | Preethi Jyothi
Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers)
Abhishek Gupta | Amruta Parulekar | Sameep Chattopadhyay | Preethi Jyothi
Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers)
Spontaneous or conversational multilingual speech presents many challenges for state-of-the-art automatic speech recognition (ASR) systems. In this work, we present a new technique AMPS, that augments a multilingual multimodal ASR system with paraphrase-based supervision for improved conversational ASR in multiple languages, including Hindi, Marathi, Malayalam, Kannada, and Nyanja. We use paraphrases of the reference transcriptions as additional supervision while training the multimodal ASR model and selectively invoke this paraphrase objective for utterances with poor ASR performance. Using AMPS with a state-of-the-art multimodal model SeamlessM4T, we obtain significant relative reductions in word error rates (WERs) of up to 5%. We present detailed analyses of our system using both objective and human evaluation metrics.
2024
Parameter-efficient Adaptation of Multilingual Multimodal Models for Low-resource ASR
Abhishek Gupta | Amruta Parulekar | Sameep Chattopadhyay | Preethi Jyothi
Proceedings of the Fourth Workshop on Multilingual Representation Learning (MRL 2024)
Abhishek Gupta | Amruta Parulekar | Sameep Chattopadhyay | Preethi Jyothi
Proceedings of the Fourth Workshop on Multilingual Representation Learning (MRL 2024)
Automatic speech recognition (ASR) for low-resource languages remains a challenge due to the scarcity of labeled training data. Parameter-efficient fine-tuning and text-only adaptation are two popular methods that have been used to address such low-resource settings. In this work, we investigate how these techniques can be effectively combined using a multilingual multimodal model like SeamlessM4T. Multimodal models are able to leverage unlabeled text via text-only adaptation with further parameter-efficient ASR fine-tuning, thus boosting ASR performance. We also show cross-lingual transfer from a high-resource language, achieving up to a relative 17% WER reduction over baseline in an extremely low-resource setting without any labeled speech.