Jiajun Deng
Author directory2026
UNISON: A Unified Sound Generation and Editing Framework via Deep LLM Fusion
Zhaoqing Li | Haoning Xu | Jingran Su | Yaofang Liu | Zhefan Rao | Huimeng Wang | Jiajun Deng | Tianzi Wang | Zengrui Jin | Rui Liu | Haoxuan Che | Xunying Liu
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Zhaoqing Li | Haoning Xu | Jingran Su | Yaofang Liu | Zhefan Rao | Huimeng Wang | Jiajun Deng | Tianzi Wang | Zengrui Jin | Rui Liu | Haoxuan Che | Xunying Liu
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
We present UNISON, a latent flow-matching framework that unifies speech generation, sound generation, and audio editing within a single model. A single model handles text-to-audio, text-to-speech, zero-shot speaker cloning, mixed speech-and-sound generation, scene-level audio editing, speech-in-scene editing, and timed audio composition, all of which share a single set of weights. Our architecture features two core designs: (1) Layer-wise deep LLM fusion, which injects hidden states from uniformly sampled layers of a frozen MLLM into corresponding MM-DiT blocks via learned projections, providing depth-varying semantic conditioning. This configuration achieves better T2A FD and CLAP than a persistent penultimate-layer conditioning baseline; and (2) a unified multi-task architecture where task identity is encoded solely by a channel-wise mask and source audio is provided through VAE-encoded channel concatenation. Training is stabilized by an online GPU-side multi-task data synthesis pipeline with task-homogeneous batching and a two-stage curriculum. With 621M–732M trainable generative parameters, UNISON achieves competitive results across the evaluated domains. Its trainable generative backbone uses 1.8–4.9x fewer parameters than the closest unified systems considered, while deployment additionally requires a frozen 7B text encoder. Audio samples and source code are publicly available.