Spoken dialogue models currently lack the ability for fine-grained speech style control, a critical capability for human-like interaction that is often overlooked in favor of purely functional capabilities like reasoning and question answering. To address this limitation, we introduce **UltraVoice**, the first large-scale spoken dialogue dataset engineered for multiple fine-grained speech style control. Encompassing over 830 hours of spoken dialogues, UltraVoice provides instructions across six key speech stylistic dimensions: emotion, speed, volume, accent, language, and composite styles. Fine-tuning leading models such as SLAM-Omni and VocalNet on UltraVoice significantly enhances their fine-grained speech stylistic controllability without degrading core conversational abilities. Specifically, our fine-tuned models achieve improvements of 29.12-42.33% in Mean Opinion Score (MOS) and 14.61-40.09 percentage points in Instruction Following Rate (IFR) on multi-dimensional control tasks designed in the UltraVoice. Moreover, on the URO-Bench benchmark, our fine-tuned models demonstrate substantial gains in core understanding, reasoning, and conversational abilities, with average improvements of +10.84% on the Basic setting and +7.87% on the Pro setting. Furthermore, the dataset’s utility extends to training controllable Text-to-Speech (TTS) models, underscoring its high quality and broad applicability for expressive speech synthesis. The complete dataset and model checkpoints are available at: [https://github.com/bigai-nlco/UltraVoice](https://github.com/bigai-nlco/UltraVoice).

Inspired by Large Language Models (LLMs), codec-based TTS systems have achieved remarkable success in high-fidelity speech generation by directly adopting the standard Cross-Entropy (CE) objective to predict sequences of discrete speech tokens from neural audio codecs. However, this practice overlooks the geometric structure of codec token spaces: tokens that are close in the codec codebook embedding space often correspond to acoustically similar speech realizations and can serve as perceptually plausible alternatives. We term this observation the Neighboring Token Phenomenon. The one-hot CE objective is agnostic to such structure, as it penalizes all non-target tokens equally, including these close neighbors, creating a mismatch between the structured codec latent space and the learning objective. In this paper, we quantitatively analyze the neighboring token phenomenon in both reconstruction and generation settings. To address the mismatch, we propose Multi-Neighbor Cross-Entropy (MNCE) loss function, which generalizes the one-hot target of CE loss into a positive set containing the ground-truth token and its nearest neighbors, encouraging the model to treat neighbor predictions as partially correct. As a result, even when the model misses the exact target token, it is more likely to predict a close neighbor rather than an unrelated token, reducing perceptually harmful errors in synthesized speech. Experiments demonstrate that MNCE is effective and generalizable across diverse codec-based TTS systems, improving quality and robustness. Moreover, scaling training to 20,000 hours further validates the scalability of our approach.

Evaluating expressive speech remains challenging, as existing methods mainly assess emotional intensity and overlook whether a speech sample is expressively appropriate for its contextual setting. This limitation hinders reliable evaluation of speech systems used in narrative-driven and interactive applications, such as audiobooks and conversational agents. We introduce CEAEval, a Context-rich framework for Evaluating Expressive Appropriateness in speech, which assesses whether a speech sample expressively aligns with the underlying communicative intent implied by its discourse-level narrative context. To support this task, we construct CEAEval-D, the first context-rich speech dataset with real human performances in Mandarin conversational speech, providing narrative descriptions together with fifteen dimensions of human annotations covering expressive attributes and expressive appropriateness. We further develop CEAEval-M, a model that integrates knowledge distillation, planner-based multi-model collaboration, adaptive audio attention bias, and reinforcement learning to perform context-rich expressive appropriateness evaluation. Experiments on a human-annotated test set demonstrate that CEAEval-M substantially outperforms existing speech evaluation and analysis systems.

Modeling fine-grained speaking styles remains challenging for language-speech representation pre-training, as existing speech-text models are typically trained with coarse captions or task-specific supervision, and scalable fine-grained style annotations are unavailable. We present FCaps, a large-scale dataset with fine-grained free-text style descriptions, encompassing 47k hours of speech and 19M fine-grained captions annotated via a novel end-to-end pipeline that directly grounds detailed captions in audio, thereby avoiding the error propagation caused by LLM-based rewriting in existing cascaded pipelines. Evaluations using LLM-as-a-judge demonstrate that our annotations surpass existing cascaded annotations in terms of correctness, coverage, and naturalness. Building on FCaps, we propose CLSP, a contrastive language-speech pre-trained model that integrates global and fine-grained supervision, enabling unified representations across multiple granularities. Extensive experiments demonstrate that CLSP learns fine-grained and multi-granular speech-text representations that perform reliably across global and fine-grained speech-text retrieval, zero-shot paralinguistic classification, and speech style similarity scoring, with strong alignment to human judgments. Code and dataset are publicly available at https://github.com/yfyeung/CLSP.