Despite advancements in controllable text-to-music (TTM) generation, existing models struggle with cross-domain voice cloning for singing due to inadequate modeling of speech-derived vocal characteristics, inherently limiting precise timbre transfer. To achieve precise and controllable music generation conditioned on the timbre of reference speech, we propose S2M-Inject, a cross-domain framework driven by reference speech. Incorporating a fixed timbre embedding strategy, S2M-Inject injects speaker representations into an instruction-guided generation pipeline. This effectively preserves target vocal attributes while enabling comprehensive natural-language control over lyrics, genre, instrumentation, rhythm, melody, and mood. Extensive experiments demonstrate that the framework achieves effective cross-domain voice cloning, significantly enhancing vocal similarity while maintaining high-fidelity audio quality in both Chinese and English tasks. This work extends voice cloning into the musical domain, offering a rigorous paradigm for personalized music generation. Audio samples are available at https://anonymous.4open.science/w/S2M-Inject-975.

Zero-shot text-to-speech (TTS) aims to synthesize high-quality speech for unseen speakers using only a short reference utterance. Existing non-autoregressive TTS models typically convert text tokens into speech-length representations through filler-token padding, explicit duration prediction, or text upsampling. However, these approaches either introduce redundant computation or rely on explicit alignment assumptions. We propose M3-TTS, a native two-stream multi-modal diffusion Transformer that preserves the original lengths of text and speech representations and enables direct text–speech interaction through packed joint self-attention. M3-TTS employs a dual-stage architecture consisting of Joint-DiT layers for learning text–speech correspondence and audio-only DiT layers for acoustic refinement. Reference acoustic information is injected through AdaLN-style conditioning. Experiments on the Seed-TTS benchmark demonstrate strong intelligibility and competitive speaker similarity and perceptual quality. The Mel-VAE variant further improves inference efficiency while maintaining high synthesis quality. Code and demos will be available at https://github.com/WWWWxp/M3-TTS.

Speech codecs that convert continuous speech signals into discrete tokens have become essential for speech language models. However, existing codecs struggle to balance high-quality reconstruction with semantically rich representations, limiting their effectiveness in both generative and understanding tasks. In this work, we propose SAC, a neural speech codec with semantic-acoustic dual-stream quantization. By disentangling semantic and acoustic modeling into two dedicated streams, SAC enables each to be optimized for its respective role. Comprehensive evaluations show that SAC achieves strong reconstruction performance across diverse bitrates under both clean and noisy conditions, with particularly high scores on UTMOS and WER, indicating superior naturalness and intelligibility. Moreover, SAC substantially surpasses prior codecs in semantic representation, approaching the level of continuous self-supervised embeddings. When used as a tokenizer for LLM-based text-to-speech, SAC enables a single-stage autoregressive (AR) TTS model that clearly outperforms state-of-the-art AR systems. Our disentanglement analysis further validates the effectiveness of the dual-stream design, offering new potential for controllable speech generation.

Generative audio modeling has largely been fragmented into specialized tasks, text-to-speech (TTS), text-to-music (TTM), and text-to-audio (TTA), each operating under heterogeneous control paradigms. Unifying these modalities remains a fundamental challenge due to the intrinsic dissonance between structured semantic representations (speech/music) and unstructured acoustic textures (sound effects). In this paper, we introduce UniSonate, a unified flow-matching framework capable of synthesizing speech, music, and sound effects through a standardized, reference-free natural language instruction interface. To reconcile structural disparities, we propose a novel dynamic token injection mechanism that projects unstructured environmental sounds into a structured temporal latent space, enabling precise duration control within a phoneme-driven Multimodal Diffusion Transformer (MM-DiT). Coupled with a multi-stage curriculum learning strategy, this approach effectively mitigates cross-modal optimization conflicts. Extensive experiments demonstrate that UniSonate achieves state-of-the-art performance in instruction-based TTS (WER 1.47%) and TTM (SongEval Coherence 3.18), while maintaining competitive fidelity in TTA. Crucially, we observe positive transfer, where joint training on diverse audio data significantly enhances structural coherence and prosodic expressiveness compared to single-task baselines.

This paper introduces F5-TTS, a fully non-autoregressive text-to-speech system based on flow matching with Diffusion Transformer (DiT). Without requiring complex designs such as duration model, text encoder, and phoneme alignment, the text input is simply padded with filler tokens to the same length as input speech, and then the denoising is performed for speech generation, which was originally proved feasible by E2 TTS. However, the original design of E2 TTS makes it hard to follow due to its slow convergence and low robustness. To address these issues, we first model the input with ConvNeXt to refine the text representation, making it easy to align with the speech. We further propose an inference-time Sway Sampling strategy, which significantly improves our model’s performance and efficiency. This sampling strategy for flow step can be easily applied to existing flow matching based models without retraining. Our design allows faster training and achieves an inference RTF of 0.15, which is greatly improved compared to state-of-the-art diffusion-based TTS models. Trained on a public 100K hours multilingual dataset, our F5-TTS exhibits highly natural and expressive zero-shot ability, seamless code-switching capability, and speed control efficiency. We have released all codes and checkpoints to promote community development, at https://SWivid.github.io/F5-TTS/.