Sashuai Zhou
Author directory2026
One-Stage Multi-Task Instruction-Guided 3D Spatial Audio Editing
Ke Lei | Chenyuhao Wen | Yu Zhang | Wenxiang Guo | Changhao Pan | Sashuai Zhou | Yongshi Li | Ruiqi Li | Ruofan Hu | Haorui Xu | Xiang Yin | Zhou Zhao
Findings of the Association for Computational Linguistics: EMNLP 2026
Ke Lei | Chenyuhao Wen | Yu Zhang | Wenxiang Guo | Changhao Pan | Sashuai Zhou | Yongshi Li | Ruiqi Li | Ruofan Hu | Haorui Xu | Xiang Yin | Zhou Zhao
Findings of the Association for Computational Linguistics: EMNLP 2026
Spatial audio editing modifies an existing soundfield according to a user’s instruction while preserving the rest of the scene. Unlike conventional audio editing, it must reason jointly about audio events, spatial information, dynamic changes, and environmental information in first-order Ambisonic (FOA) waveforms. Existing language-guided editors mainly target conventional audio or rely on sequential operations, and therefore do not directly support one-stage editing for complex 3D spatial instructions. We present SwanWeave, the first one-stage multi-task framework for instruction-guided 3D FOA spatial audio editing. We build paired FOA supervision from open-source speech and sound-effect corpora using controllable room simulation, covering more than ten single-operation and compound tasks across the four editing axes. To handle this heterogeneous edit space, SwanWeave uses Spatial Edit Mixture-of-Experts (SE-MoE) with dual-level routing, selecting task-aware expert combinations for compound instructions and frame-level routed/null experts for local edit decisions. We further introduce Spatial Preference Optimization (SPO), a Direct Preference Optimization (DPO)-based alignment objective with edit-specific negative targets, and adopt staged training to improve natural-language grounding. Experiments show that SwanWeave achieves better editing quality than existing general audio editors and spatial audio baselines across all tasks. Spatial audio editing demos can be found at https://swanaigc.github.io/#swanweave, code can be found at: https://github.com/MM-Speech/SwanWeave.
Doc-REFRAG: Rethinking Multimodal Document Retrieval-Augmented Generation
Ruofan Hu | Shengyang Xu | Minjie Hong | Xiaoda Yang | Sashuai Zhou | Ke Lei | Tao Jin | Zhou Zhao
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Ruofan Hu | Shengyang Xu | Minjie Hong | Xiaoda Yang | Sashuai Zhou | Ke Lei | Tao Jin | Zhou Zhao
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Real-world knowledge resides in multimodal documents, necessitating retrieval-augmented generation (RAG) for accurate question answering. However, existing multimodal RAG models are primarily designed for single-image or closed-document settings and exhibit limited accuracy in realistic multi-image scenarios. Moreover, processing numerous retrieved images incurs substantial computational overhead from irrelevant visual tokens. To address these challenges, we introduce DocLongRAG, a large-scale dataset of 343K question–answer pairs, each associated with an average of 37.4 retrieved images to reflect authentic RAG workflows. Building on this dataset, we propose Doc-REFRAG, a question-guided framework that compresses visual tokens into coarse chunks and selectively expands question-relevant ones via a lightweight RL-based selector. Experiments on six benchmarks show that Doc-REFRAG outperforms eleven strong baselines, achieving state-of-the-art accuracy with significantly lower inference latency. Our resources are available at https://github.com/Collab-Gen/Doc-REFRAG.
Unified Thinker: A General Reasoning Core for Image Generation
Sashuai Zhou | Qiang Zhou | Jijin Hu | Hanqing Yang | Yue Cao | Junpeng Ma | Yinchao Ma | Jun Song | Tiezheng Ge | Cheng Yu | Bo Zheng | Zhou Zhao
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Sashuai Zhou | Qiang Zhou | Jijin Hu | Hanqing Yang | Yue Cao | Junpeng Ma | Yinchao Ma | Jun Song | Tiezheng Ge | Cheng Yu | Bo Zheng | Zhou Zhao
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Despite impressive progress in high-fidelity image synthesis, generative models still struggle with logic-intensive instruction following, exposing a persistent reasoning–execution gap. Meanwhile, closed-source systems (e.g., Nano Banana) have demonstrated strong reasoning-driven image generation, highlighting a substantial gap to current open-source models. We argue that closing this gap requires not merely better visual generators, but executable reasoning: decomposing high-level intents into grounded, verifiable plans that directly steer the generative process. To this end, we propose Unified Thinker, a task-agnostic reasoning architecture for general image generation, designed as a unified planning core that can plug into diverse generators and workflows. Unified Thinker decouples a dedicated Thinker from the image Generator, enabling modular upgrades of reasoning without retraining the entire generative model. We further introduce a two-stage training paradigm: we first build a structured planning interface for the Thinker, then apply reinforcement learning to ground its policy in pixel-level feedback, encouraging plans that optimize visual correctness over textual plausibility. Extensive experiments on text-to-image generation and image editing show that Unified Thinker substantially improves image reasoning and generation quality.
2025
Overcoming both Domain Shift and Label Shift for Referring Video Segmentation
Hai Huang | Sashuai Zhou | Yan Xia
Findings of the Association for Computational Linguistics: NAACL 2025
Hai Huang | Sashuai Zhou | Yan Xia
Findings of the Association for Computational Linguistics: NAACL 2025
Open-set domain generalization (OSDG) aims to enhance the robustness of the model when facing both domain shift and label shift, highlighting a wide range of potential in real-world applications. However, previous OSDG methods can only recognize seen objects and mark all unseen objects as “unknown” categories during inference, which is far from satisfactory. In this paper, we explore the scenario of referring video segmentation to study how to make the model maintain good segmentation ability for unknown objects under OSDG setting. To bridge the huge gap caused by label shift, we propose CLIP-based Reasoning Prompt (CRPrompt), which can combine text and visual prompts together to improve text-object matching ability of CLIP, transferring the segmentation ability to unseen classes based on the knowledge learned from seen classes and large-scale text-image pairs, i.e., color, shape, spatial relationships. Meanwhile, to improve the robustness of CRPrompt, we propose Retrieval-augmented Instance Normalization (RaIN), which can effectively enhance the robustness of the model by retrieving visual objects with similar semantic concepts through input query and performing Instance Norm among them. Extensive experiments on open-set and zero-shot domain generalization tasks demonstrate the effectiveness of our approach.
Enhancing Multimodal Unified Representations for Cross Modal Generalization
Hai Huang | Yan Xia | Shengpeng Ji | Shulei Wang | Hanting Wang | Minghui Fang | Jieming Zhu | Zhenhua Dong | Sashuai Zhou | Zhou Zhao
Findings of the Association for Computational Linguistics: ACL 2025
Hai Huang | Yan Xia | Shengpeng Ji | Shulei Wang | Hanting Wang | Minghui Fang | Jieming Zhu | Zhenhua Dong | Sashuai Zhou | Zhou Zhao
Findings of the Association for Computational Linguistics: ACL 2025
To enhance the interpretability of multimodal unified representations, many studies have focused on discrete unified representations. These efforts typically start with contrastive learning and gradually extend to the disentanglement of modal information, achieving solid multimodal discrete unified representations. However, existing research often overlooks two critical issues: 1) The use of Euclidean distance for quantization in discrete representations often overlooks the important distinctions among different dimensions of features, resulting in redundant representations after quantization; 2) Different modalities have unique characteristics, and a uniform alignment approach does not fully exploit these traits. To address these issues, we propose Training-free Optimization of Codebook (TOC) and Fine and Coarse cross-modal Information Disentangling (FCID). These methods refine the unified discrete representations from pretraining and perform fine- and coarse-grained information disentanglement tailored to the specific characteristics of each modality, achieving significant performance improvements over previous state-of-the-art models. The code is available at https://github.com/haihuangcode/CMG.
RecBase: Generative Foundation Model Pretraining for Zero-Shot Recommendation
Sashuai Zhou | Weinan Gan | Qijiong Liu | Ke Lei | Jieming Zhu | Hai Huang | Yan Xia | Ruiming Tang | Zhenhua Dong | Zhou Zhao
Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing
Sashuai Zhou | Weinan Gan | Qijiong Liu | Ke Lei | Jieming Zhu | Hai Huang | Yan Xia | Ruiming Tang | Zhenhua Dong | Zhou Zhao
Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing
Recent advances in LLM-based recommendation have shown promise, yet their cross-domain generalization is hindered by a fundamental mismatch between language-centric pretraining and the recommendation task. Existing methods, relying on language-level knowledge, fail to capture dynamic, item-level user interests across domains. To bridge this gap, we propose RecBase, a domain-agnostic foundational model pretrained with a recommendation-oriented objective. RecBase leverages a large-scale, heterogeneous, cross-domain corpus with unified textual representations and feature mappings to enhance cross-domain generalization. To further align item semantics across domains, we introduce a unified item tokenizer that encodes items into hierarchical concept identifiers, enabling structured representation and efficient vocabulary sharing. The model is trained using an autoregressive objective to capture complex item-level sequential patterns. On eight real-world datasets, our 1.5B-parameter model matches or surpasses the performance of LLM baselines up to 7B parameters in zero-shot and cross-domain recommendation tasks.
T2A-Feedback: Improving Basic Capabilities of Text-to-Audio Generation via Fine-grained AI Feedback
Zehan Wang | Ke Lei | Chen Zhu | Jiawei Huang | Sashuai Zhou | Luping Liu | Xize Cheng | Shengpeng Ji | Zhenhui Ye | Tao Jin | Zhou Zhao
Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Zehan Wang | Ke Lei | Chen Zhu | Jiawei Huang | Sashuai Zhou | Luping Liu | Xize Cheng | Shengpeng Ji | Zhenhui Ye | Tao Jin | Zhou Zhao
Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Text-to-audio (T2A) generation has achieved remarkable progress in generating a variety of audio outputs from language prompts. However, current state-of-the-art T2A models still struggle to satisfy human preferences for prompt-following and acoustic quality when generating complex multi-event audio. To improve the performance of the model in these high-level applications, we propose to enhance the basic capabilities of the model with AI feedback learning. First, we introduce fine-grained AI audio scoring pipelines to: 1) verify whether each event in the text prompt is present in the audio (Event Occurrence Score), 2) detect deviations in event sequences from the language description (Event Sequence Score), and 3) assess the overall acoustic and harmonic quality of the generated audio (Acoustic&Harmonic Quality). We evaluate these three automatic scoring pipelines and find that they correlate significantly better with human preferences than other evaluation metrics. This highlights their value as both feedback signals and evaluation metrics. Utilizing our robust scoring pipelines, we construct a large audio preference dataset, T2A-FeedBack, which contains 41k prompts and 249k audios, each accompanied by detailed scores. Moreover, we introduce T2A-EpicBench, a benchmark that focuses on long captions, multi-events, and story-telling scenarios, aiming to evaluate the advanced capabilities of T2A models. Finally, we demonstrate how T2A-FeedBack can enhance current state-of-the-art audio model. With simple preference tuning, the audio generation model exhibits significant improvements in both simple (AudioCaps test set) and complex (T2A-EpicBench) scenarios.
Search
Fix author
Co-authors
- Zhou Zhao 6
- Ke Lei 4
- Hai Huang 3
- Yan Xia 3
- Zhenhua Dong 2
- Ruofan Hu 2
- Shengpeng Ji 2
- Tao Jin 2
- Jieming Zhu 2
- Yue Cao 1
- Xize Cheng 1
- Minghui Fang 1
- Weinan Gan 1
- Tiezheng Ge 1
- Wenxiang Guo 1
- Minjie Hong 1
- Jijin Hu 1
- Jiawei Huang 1
- Ruiqi Li 1
- Yongshi Li 1
- Luping Liu 1
- Qijiong Liu 1
- Junpeng Ma 1
- Yinchao Ma 1
- Changhao Pan 1
- Jun Song 1
- Ruiming Tang 1
- Hanting Wang 1
- Shulei Wang 1
- Zehan Wang 1
- Chenyuhao Wen 1
- Haorui Xu 1
- Shengyang Xu 1
- Hanqing Yang 1
- Xiaoda Yang 1
- Zhenhui Ye 1
- Xiang Yin 1
- Cheng Yu 1
- Yu Zhang 1
- Bo Zheng 1
- Qiang Zhou 1
- Chen Zhu 1