Ziyu Zhang
Author directory2026
S2M-Inject: Prompt-Guided Zero-Shot Voice Cloning for Music Generation
Teng Ma | Chunyu Qiang | Zhongliang Liu | Xiaopeng Wang | Kang Yin | Ziyu Zhang | Yuzhe Liang | Yushen Chen | Yuxin Guo | Feiyu Chen
Findings of the Association for Computational Linguistics: EMNLP 2026
Teng Ma | Chunyu Qiang | Zhongliang Liu | Xiaopeng Wang | Kang Yin | Ziyu Zhang | Yuzhe Liang | Yushen Chen | Yuxin Guo | Feiyu Chen
Findings of the Association for Computational Linguistics: EMNLP 2026
Despite advancements in controllable text-to-music (TTM) generation, existing models struggle with cross-domain voice cloning for singing due to inadequate modeling of speech-derived vocal characteristics, inherently limiting precise timbre transfer. To achieve precise and controllable music generation conditioned on the timbre of reference speech, we propose S2M-Inject, a cross-domain framework driven by reference speech. Incorporating a fixed timbre embedding strategy, S2M-Inject injects speaker representations into an instruction-guided generation pipeline. This effectively preserves target vocal attributes while enabling comprehensive natural-language control over lyrics, genre, instrumentation, rhythm, melody, and mood. Extensive experiments demonstrate that the framework achieves effective cross-domain voice cloning, significantly enhancing vocal similarity while maintaining high-fidelity audio quality in both Chinese and English tasks. This work extends voice cloning into the musical domain, offering a rigorous paradigm for personalized music generation. Audio samples are available at https://anonymous.4open.science/w/S2M-Inject-975.
UniSonate: A Unified Model for Speech, Music, and Sound Effect Generation with Text Instructions
Chunyu Qiang | Xiaopeng Wang | Kang Yin | Yuzhe Liang | Yuxin Guo | Teng Ma | Ziyu Zhang | Tianrui Wang | Cheng Gong | Yushen Chen | Ruibo Fu | Longbiao Wang | Jianwu Dang
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Chunyu Qiang | Xiaopeng Wang | Kang Yin | Yuzhe Liang | Yuxin Guo | Teng Ma | Ziyu Zhang | Tianrui Wang | Cheng Gong | Yushen Chen | Ruibo Fu | Longbiao Wang | Jianwu Dang
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Generative audio modeling has largely been fragmented into specialized tasks, text-to-speech (TTS), text-to-music (TTM), and text-to-audio (TTA), each operating under heterogeneous control paradigms. Unifying these modalities remains a fundamental challenge due to the intrinsic dissonance between structured semantic representations (speech/music) and unstructured acoustic textures (sound effects). In this paper, we introduce UniSonate, a unified flow-matching framework capable of synthesizing speech, music, and sound effects through a standardized, reference-free natural language instruction interface. To reconcile structural disparities, we propose a novel dynamic token injection mechanism that projects unstructured environmental sounds into a structured temporal latent space, enabling precise duration control within a phoneme-driven Multimodal Diffusion Transformer (MM-DiT). Coupled with a multi-stage curriculum learning strategy, this approach effectively mitigates cross-modal optimization conflicts. Extensive experiments demonstrate that UniSonate achieves state-of-the-art performance in instruction-based TTS (WER 1.47%) and TTM (SongEval Coherence 3.18), while maintaining competitive fidelity in TTA. Crucially, we observe positive transfer, where joint training on diverse audio data significantly enhances structural coherence and prosodic expressiveness compared to single-task baselines.