Instruction-guided text-to-speech (TTS) systems are among the most intuitive for controllable speech synthesis, but writing descriptive instructions in the form of style prompts is too cumbersome for users to maintain natural conversation rhythm. We introduce TurnStyle, a prompt generation system, that produces style prompts for instruction-guided TTS, leveraging a bimodal dialogue history (speech and text transcripts) and a user script query. To address the unintuitive interface problem, TurnStyle creates a style prompt by conditioning the Llama-3.2-3B-Instruct model on a prefix vector generated from an embedded bimodal representation of a dialogue. Computational experiments with performance measured across standard metrics like BERTscore and ROUGE-L show that TurnStyle’s architecture improved on a comparative baseline model for the understudied task of style prompt generation. The introduced model also consumed over 15 times less energy than the baseline and required less memory, suggesting that the TurnStyle framework may feasibly run on edge devices.

As Speech Large Language Models (speech-LLMs) become increasingly multilingual, Arabic remains significantly underrepresented, highlighting the need for dedicated infrastructure to train and evaluate Arabic speech-LLMs. To address this gap, we introduce Nuha-Speech, a comprehensive initiative to develop general-purpose Arabic speech-LLMs spanning dataset construction, model training, and systematic evaluation. Specifically, we constructed a large-scale Arabic Speech Question-Answering (SQA) corpus comprising over 1.5 million training samples to allow instruction tuning over a broad range of core speech tasks. Then, the corpus was used for supervised fine-tuning based on Qwen-Omni model variants at different scales. Finally, we designed an evaluation framework featuring diverse tasks and tailored metrics. Through this work, we aim to establish foundational infrastructures for Arabic Speech-LLMs under constraints imposed by limited Arabic speech resources.

Audio-Language Models (ALMs) natively fuse lexical content with continuous paralinguistic acoustics such as intonation, tone, and speaker identity. While considerable progress has been made in aligning the semantic outputs of text-based models against harmful instructions, current ALM safety alignment evaluation largely treats speech as a noisy text channel, leaving modality-specific vulnerabilities unexamined. In this work, we introduce and investigate Prosody-Conditioned Safety Divergence: a vulnerability where an ALM’s safety behavior shifts systematically based entirely on paralinguistic variation, even when lexical semantic payload is held completely constant. Rather than assuming the existence of coarse, high-variance acoustic "authority" subspaces, we conduct a controlled causal analysis to track how specific prosodic manipulations—including variations in pitch contour and intensity—interact with safety-associated latent directions across the encoder, projector, and decoder layers. Our preliminary empirical evaluations characterize the localization of this divergence, establishing a framework for targeted inference-time interventions that restore alignment metrics without compromising baseline utility on benign speech tasks.

Verbatim transcripts, which keep the hesitations and interactional particles that ordinary automatic speech recognition (ASR) removes, are a prerequisite for speech-based cognitive assessment, and they are hardest to obtain for languages with little annotated speech. We study this problem for Swedish with a 1.02-hour corpus of manually transcribed spontaneous speech, too small for full fine-tuning of KB-Whisper-large. We adapt the model with a low-rank adapter and compare three ways of extending the training data: pseudo-labeling of untranscribed in-domain audio, unguided synthesis with a language model and a text-to-speech system, and a gold-preserving pipeline that reproduces the disfluency distribution of the gold corpus by construction and can be audited before training. Fine-tuning reduces disfluency-segment word error rate (WER) by 32% relative, with blockwise-bootstrap confidence intervals disjoint from the zero-shot baseline, while the three augmentation strategies are not separable by WER across five seeds. The gold-preserving pipeline needs no untranscribed audio, and its disfluency distribution can be verified in advance.

Artificial intelligence (AI) assistants for blind and low-vision (BLV) users deliver scene descriptions through a serial auditory channel, often followed by spoken user queries. A central challenge is that scene-description pipelines typically prioritize visually or corpus-salient content, whereas users’ follow-up questions often depend on low-salience attributes that are omitted during generation or compression. To diagnose where this mismatch arises, we introduce an encoder-agnostic framework that decomposes scene-description systems into four stages: visual encoding, length control, content selection, and audio scheduling. Applying this framework to 200 blind-user-captured VizWiz images and 500 questions reveals a salience–relevance mismatch: salience-based compression preserves identity information but discards low-salience attributes needed for attribute-bound questions. Improving the visual-encoding stage by replacing BLIP-base with stronger open-weight MLLMs raises semantic retention but does not eliminate this mismatch. As a lightweight repair, we propose CA-Keyword, which explicitly preserves attribute terms and recovers color-question answerability by 0.23 while producing descriptions 37% shorter than the sentence/

While text-based hallucination detection is well studied, reference-free detection of factual alterations in speech remains underexplored, especially for low-resource languages. Our spoken benchmark comprises 12,013 English, Russian, and Kazakh news samples with three synthetic alteration types and three severity levels, pairing source articles with rewrites as text, synthesized audio, and ASR transcripts. We add 290 fact-checked misinformation items collected in Russian (225) and Kazakh (65), translated into the other language and rendered through the same TTS–ASR pipeline. We evaluate fine-tuned multilingual encoders and zero-shot multimodal decoders on text, transcripts, and audio. Detectors receive only target inputs without source articles or external evidence; the task evaluates reference-free classification rather than evidence-grounded verification. Encoder degradation from source text to transcripts generally tracks per-language ASR error on the binary task. Among decoders, only Gemma-3n exceeds the binary majority-class baseline in macro-F1, on transcripts only; the other four do not exceed their respective baselines. Comparisons between audio and transcripts are confounded by differences in evaluation coverage and class balance. Synthetic-trained detectors achieve 0.82–0.88 macro-F1 on real-world misinformation source text; Russian provenance analysis reveals veracity-related and model-dependent machine-style signals, a key confound in synthetic hallucination benchmarks.

Current multilingual automatic speech recognition (ASR) models have substantially improved speech recognition across many languages, yet recognition quality remains limited for low-resource languages such as Urdu. A common approach to improve recognition quality is to fine-tune the ASR model itself, but this requires labelled speech data and retraining the recognizer. A recently emerging alternative is Generative Error Correction (GEC), in which a language model post-processes ASR output to correct transcription errors while leaving the underlying speech recognizer unchanged. This paper investigates this approach for Urdu by keeping Whisper-Turbo frozen and fine-tuning small language models to correct its transcription errors. To address data scarcity in Urdu, we construct a pseudo-gold training dataset using a strong teacher model. Fine-tuning 3.5–4B parameter language models on this data achieves a ~33% relative WER reduction (from 11.74% to 7.81%), outperforming zero-shot baselines as well as significantly larger 12–14B zero-shot models. These results show that combining a frozen recognizer with a small fine-tuned corrector provides a practical, resource-efficient framework for Urdu ASR improvement without relying on ASR retraining or substantially larger models.

Modern audio-language models are no longer judged only on what words they can transcribe, but on whether they can reason over what they hear: recovering meaning that lives in tone and prosody, telling dialects and regional languages apart, and resolving ambiguity that the written form leaves open. This capability is now measured by a growing family of audio-reasoning benchmarks, but almost entirely in English and on general-domain audio. Southeast Asia (SEA) is served instead by benchmarks that inherit an English task taxonomy of recognition, translation, and paralinguistic classification, and therefore test whether a model hears SEA speech rather than whether it can reason from it. We introduce SEABED, an audio-first question answering dataset designed to benchmark language and audio reasoning models on SEA speech. SEABED comprises a suite of six audio-reasoning tasks built entirely from real, openly available SEA speech corpora, yielding 5,404 question-answer pairs. We evaluate six frontier and region-specific audio LLMs: even the state-of-the-art model Gemini 3.5 Flash achieves only 50.3% weighted average accuracy. SEABED evaluates not only answer accuracy, but also whether models’ stated reasoning is grounded in the audio evidence. A sample of the benchmark data is available here: https://huggingface.co/datasets/CentificAIResearch/SEABED.

The functional organization of prosody within the Residual Vector Quantization (RVQ) codebooks of neural audio codecs remains an open question. Prior work relies on correlational probing, showing that pitch features can be decoded from frozen codebook embeddings. Decodability, however, does not imply causal necessity: a codebook may encode pitch information the decoder ignores, while another may be essential despite containing little probe-recoverable signal. We address this gap through causal output-side ablation: for each codebook, we zero or mean-replace its discrete tokens, re-decode the waveform, and measure degradation in fundamental frequency F₀ (CREPE), energy (frame-wise RMS), and intelligibility (STOI) against the unablated reconstruction. Experiments span 200 LibriSpeech test-clean utterances under clean, 15, 5, and 0 dB SNR with MUSAN noise across multiple neural audio codecs. Across different architectures, we find a consistent two-tier causal structure: a codec-specific subset of codebooks is reconstruction-critical, while the remaining intelligibility-preserving codebooks exhibit a stable pitch-contribution hierarchy that persists under additive noise. These findings provide new evidence for how neural audio codecs causally organize prosodic information.

We investigate representational alignment between nine independently trained speech encoders and ten text encoders. We build a controlled corpus in which the same sentences are rendered by different speakers and under different emotional conditions, so that prosody varies while linguistic content is held fixed. We compare every speech–text layer pair and calibrate the strongest match with a permutation procedure that prevents the search over pairs from inflating alignment scores. Local neighborhood agreement between modalities survives calibration throughout and is strongest between late speech layers and early text layers, providing strong evidence of genuine local alignment. Global similarity measures, however, do not provide reliable evidence that the two modalities share a common geometric space as a whole. Alignment is unaffected by speaker identity and prosodic variation, so the shared structure is tied to the content of the sentence itself rather than to how it is spoken. It also does not increase with model capability or with how well a text model reproduces human judgments of meaning, suggesting that this shared structure corresponds more closely to lexical and surface-level representations than to fine-grained sentence meaning. These findings indicate where speech and text representations can be connected, suggest that cross-modal transfer may benefit from exploiting neighborhood structure rather than assuming a shared global geometry, and provide new alignment methods with an existing lexical correspondence to build on, while pointing to sentence-level meaning as the structure that future speech representations could capture so that alignment extends beyond the lexical level.

Code-switching (CS), a seamless alternation between languages within a single utterance, remains a critical challenge in automatic speech recognition (ASR). While prior works focus on conversational CS-ASR, enterprise settings demand evaluation of operational impact beyond edit-distance errors: how code-switching transcription errors propagate to downstream voice agent task failures. In this work, we propose (1) a CS-ASR synthetic benchmark and multidimensional evaluation framework tailored to enterprise domains, (2) systematic evaluation of frontier ASR systems across 5 language pairs, (3) diagnostic analysis of the additional transcription errors that code-switching introduces across language pairs and models. We release CoSE-E to support enterprise-focused CS-ASR evaluation for multilingual voice agents in enterprise deployment.

Spoken factual claims often occur within multi-turn conversations, where surrounding dialogue can provide context unavailable from the claim alone. Yet most fact-checking research evaluates isolated text, leaving conversational audio under-studied. We introduce MAD2, a synthetic Multi-turn Audio Dialogues benchmark for spoken claim verification with 1,000 two-speaker dialogues, 1,230 sentence-level check-worthy candidate annotations, and approximately 10 hours of audio. We also propose calibrated multimodal fusion of a context-aware audio encoder and a dialogue-aware text model. Adding dialogue context improves verification across settings, although the gains vary by scenario. Past-only context often approaches local offline performance, suggesting its usefulness when future context is unavailable. Fusion achieves its highest mean advantage over text with full-dialogue context, but does not consistently or significantly outperform text across settings. Under full-dialogue context, exploratory subgroup results show greater performance variation across dialogue scenarios for text and fusion, but across spread styles for audio.

Bengali, spoken by over 230 million people, remains severely under-resourced in spoken language research. Existing open Bengali ASR corpora are confined to short, read or crowd-sourced speech and lack speaker diarization and long-form audio. We present Lipi-Ghor, a large-scale weakly-supervised Bengali speech corpus comprising 885 hours of audio sourced from 998 YouTube videos across 596 unique channels, spanning 17 diverse domains including audiobooks, drama, religious sermons, news, and regional dialects. Each recording is annotated in SSTT format (Speaker, Speech, Transcription, Timestamp), which combines automated diarization with Bengali YouTube caption tracks. To our knowledge, Lipi-Ghor is the first Bengali speech corpus to pair large-scale coverage with speaker-diarized, long-form audio annotations, both of which are absent from existing open Bengali resources. We also present the construction methodology as a replicable pipeline for any language with a sufficient captioned YouTube presence. Targeted channel curation across a handful of rich domains yields corpus scales that were previously out of reach for low-resource languages. We benchmark four widely-used ASR models under zero-shot and fine-tuned conditions. All four degrade consistently on emotive, dialectal, and long-form content. Because caption targets are noisy, we also release a human-annotated gold evaluation split, verified by 17 volunteer annotators over 30 hours of human-in-the-loop correction and stratified across all 17 domains. We release our original contributions (video-ID lists, diarization output, pipeline code, and the gold annotations) under CC-BY-4.0, and distribute caption-derived transcripts for research use only; no raw audio is redistributed.