Sameep Chattopadhyay

Author directory

2026

Modern voice-agent systems rely on streaming speech recognition models that operate under stringent latency constraints. This study shows that, due to the limited memory constraints of real-time processing, these systems are adversely impacted by conversational phenomena such as long silences and backchannels. While many agentic pipelines mitigate this by resetting state at each turn, this approach discards vital context and impairs performance at turn onsets. We propose two state-management strategies that preserve cross-utterance context to reduce onset errors. In experiments with two state-of-the-art streaming models on two spoken dialogue benchmarks, our best method yields an average of 15-21% relative WER reduction at utterance onsets.
Audio language models (LMs) are now integrated into high-stakes, decision-oriented settings: hiring pipelines, oral assessments, and language proficiency tests. Prior work in natural language processing has primarily focused on how LMs respond to socially-driven textual variation. With the growing use of LMs that natively process audio, similar robustness and fairness issues arise for speech variation. In our work, we examine how accent variation in English inputs affects audio LMs’ assessments of speakers. We evaluate six audio LMs on speech across five accent groups: American, British, Chinese, Indian, and Nigerian English. We find that several LMs assign lower speech delivery scores to Chinese- and Nigerian-accented speakers, regardless of content quality. At the individual level, delivery scores correlate negatively with acoustic distance from American English across three widely-used models (Spearman ρ = −0.35 to −0.66, all p < .05), suggesting that English speech is scored against an implicit American acoustic norm. Even though models are able to transcribe the speech in our studies, their assessment and perceptions of speakers may result in allocational harm. We release our framework and corpus to support future analyses of bias in audio LMs.

2025

Spontaneous or conversational multilingual speech presents many challenges for state-of-the-art automatic speech recognition (ASR) systems. In this work, we present a new technique AMPS, that augments a multilingual multimodal ASR system with paraphrase-based supervision for improved conversational ASR in multiple languages, including Hindi, Marathi, Malayalam, Kannada, and Nyanja. We use paraphrases of the reference transcriptions as additional supervision while training the multimodal ASR model and selectively invoke this paraphrase objective for utterances with poor ASR performance. Using AMPS with a state-of-the-art multimodal model SeamlessM4T, we obtain significant relative reductions in word error rates (WERs) of up to 5%. We present detailed analyses of our system using both objective and human evaluation metrics.

2024

Automatic speech recognition (ASR) for low-resource languages remains a challenge due to the scarcity of labeled training data. Parameter-efficient fine-tuning and text-only adaptation are two popular methods that have been used to address such low-resource settings. In this work, we investigate how these techniques can be effectively combined using a multilingual multimodal model like SeamlessM4T. Multimodal models are able to leverage unlabeled text via text-only adaptation with further parameter-efficient ASR fine-tuning, thus boosting ASR performance. We also show cross-lingual transfer from a high-resource language, achieving up to a relative 17% WER reduction over baseline in an extremely low-resource setting without any labeled speech.