Despite advancements in controllable text-to-music (TTM) generation, existing models struggle with cross-domain voice cloning for singing due to inadequate modeling of speech-derived vocal characteristics, inherently limiting precise timbre transfer. To achieve precise and controllable music generation conditioned on the timbre of reference speech, we propose S2M-Inject, a cross-domain framework driven by reference speech. Incorporating a fixed timbre embedding strategy, S2M-Inject injects speaker representations into an instruction-guided generation pipeline. This effectively preserves target vocal attributes while enabling comprehensive natural-language control over lyrics, genre, instrumentation, rhythm, melody, and mood. Extensive experiments demonstrate that the framework achieves effective cross-domain voice cloning, significantly enhancing vocal similarity while maintaining high-fidelity audio quality in both Chinese and English tasks. This work extends voice cloning into the musical domain, offering a rigorous paradigm for personalized music generation. Audio samples are available at https://anonymous.4open.science/w/S2M-Inject-975.

Zero-shot text-to-speech (TTS) aims to synthesize high-quality speech for unseen speakers using only a short reference utterance. Existing non-autoregressive TTS models typically convert text tokens into speech-length representations through filler-token padding, explicit duration prediction, or text upsampling. However, these approaches either introduce redundant computation or rely on explicit alignment assumptions. We propose M3-TTS, a native two-stream multi-modal diffusion Transformer that preserves the original lengths of text and speech representations and enables direct text–speech interaction through packed joint self-attention. M3-TTS employs a dual-stage architecture consisting of Joint-DiT layers for learning text–speech correspondence and audio-only DiT layers for acoustic refinement. Reference acoustic information is injected through AdaLN-style conditioning. Experiments on the Seed-TTS benchmark demonstrate strong intelligibility and competitive speaker similarity and perceptual quality. The Mel-VAE variant further improves inference efficiency while maintaining high synthesis quality. Code and demos will be available at https://github.com/WWWWxp/M3-TTS.

Generative audio modeling has largely been fragmented into specialized tasks, text-to-speech (TTS), text-to-music (TTM), and text-to-audio (TTA), each operating under heterogeneous control paradigms. Unifying these modalities remains a fundamental challenge due to the intrinsic dissonance between structured semantic representations (speech/music) and unstructured acoustic textures (sound effects). In this paper, we introduce UniSonate, a unified flow-matching framework capable of synthesizing speech, music, and sound effects through a standardized, reference-free natural language instruction interface. To reconcile structural disparities, we propose a novel dynamic token injection mechanism that projects unstructured environmental sounds into a structured temporal latent space, enabling precise duration control within a phoneme-driven Multimodal Diffusion Transformer (MM-DiT). Coupled with a multi-stage curriculum learning strategy, this approach effectively mitigates cross-modal optimization conflicts. Extensive experiments demonstrate that UniSonate achieves state-of-the-art performance in instruction-based TTS (WER 1.47%) and TTM (SongEval Coherence 3.18), while maintaining competitive fidelity in TTA. Crucially, we observe positive transfer, where joint training on diverse audio data significantly enhances structural coherence and prosodic expressiveness compared to single-task baselines.