Text-aligned speech tokenization methods have emerged to better align speech tokens with LLM token spaces, enabling more effective utilization of pretrained LLMs. However, they rely on offline automatic speech recognition (ASR), leading to two key limitations: (i) the need for complete utterances before tokenization, precluding real-time streaming, and (ii) vocabulary mismatch between ASR and LLMs, which reduces acoustic granularity from the subword to the word level. We introduce StreamAlign, a text-aligned speech tokenization framework that enables streaming tokenization for real-time speech–text joint modeling. StreamAlign performs online speech–text alignment by combining character-level RNN-Transducer alignment with word-level ASR guidance, mitigating ASR–LLM vocabulary mismatch while preserving recognition accuracy. A proactive word boundary classifier anticipates word completion at chunk boundaries, reducing tokenization latency from 560ms to 270 ms. On LibriSpeech, StreamAlign achieves the lowest WER and highest UTMOS among evaluated tokenizers. Furthermore, StreamAlign-SLM, a spoken language model trained on StreamAlign units, demonstrates the strongest overall performance in terms of both acoustic and semantic consistency on the SALMon and spoken StoryCloze benchmarks.

Large audio-language models (LALMs) extend language understanding into the auditory domain, yet their ability to perform low-level listening, such as pitch and duration detection, remains underexplored. However, low-level listening is critical for real-world, out-of-distribution tasks where models must reason about unfamiliar sounds based on fine-grained acoustic cues. To address this gap, we introduce the World-of-Whale benchmark (WoW-Bench) to evaluate low-level auditory perception and cognition using marine mammal vocalizations. We use marine mammal vocalizations as out-of-distribution sound events to better assess models’ low-level listening and so that the models do not rely on prior knowledge of the sound events. WoW-bench is composed of a Perception benchmark for categorizing novel sounds and a Cognition benchmark, inspired by Bloom’s taxonomy, to assess the abilities to remember, understand, apply, and analyze sound events. For the Cognition benchmark, we additionally introduce distractor questions to evaluate whether models are truly solving problems through listening rather than relying on other heuristics. Experiments with state-of-the-art LALMs show performance far below human levels, indicating a need for stronger auditory grounding in LALMs.

Effective spoken AI assistants must decide not only what to say, but also when to say it. Although appropriate AI turn-taking behavior is inherently scenario-dependent, current approaches fall short in complementary ways: human-human speech corpora capture natural timing phenomena but provide little AI-role grounding or scenario-specific norms, while heuristic or prompted synthesis methods inject turn-taking behaviors without grounding them in human preferences. We introduce **DuplexGen**, a human-calibrated framework that generates synthetic human-AI dialogues with scenario-adaptive turn-taking. DuplexGen transforms text dialogues into spoken-style transcripts, marks candidate intra-utterance action slots, and calibrates LLM predictions using a small set of slot-level human preference annotations. Across six tasks spanning cooperative and competitive interactions, DuplexGen aligns substantially more closely with human preferences than uncalibrated prompting and models trained solely on generic human-human data; furthermore, a full-duplex model trained on DuplexGen-generated data exhibits distinctive, human-preferred turn-taking behaviors across scenarios.

Spoken dialogue systems increasingly employ large language models (LLMs) to leverage their advanced reasoning capabilities. However, direct application of LLMs in spoken communication often yield suboptimal results due to mismatches between optimal textual and verbal delivery. While existing approaches adapt LLMs to produce speech-friendly outputs, their impact on reasoning performance remains underexplored. In this work, we propose Think-Verbalize-Speak, a framework that decouples reasoning from spoken delivery to preserve the full reasoning capacity of LLMs. Central to our method is verbalizing, an intermediate step that translates thoughts into natural, speech-ready text. We also introduce ReVerT, a latency-efficient verbalizer based on incremental and asynchronous summarization. Experiments across multiple benchmarks show that our method enhances speech naturalness and conciseness with minimal impact on reasoning. The project page with the dataset and the source code is available at https://yhytoto12.github.io/TVS-ReVerT.