Ruofan Hu
Author directory2026
One-Stage Multi-Task Instruction-Guided 3D Spatial Audio Editing
Ke Lei | Chenyuhao Wen | Yu Zhang | Wenxiang Guo | Changhao Pan | Sashuai Zhou | Yongshi Li | Ruiqi Li | Ruofan Hu | Haorui Xu | Xiang Yin | Zhou Zhao
Findings of the Association for Computational Linguistics: EMNLP 2026
Ke Lei | Chenyuhao Wen | Yu Zhang | Wenxiang Guo | Changhao Pan | Sashuai Zhou | Yongshi Li | Ruiqi Li | Ruofan Hu | Haorui Xu | Xiang Yin | Zhou Zhao
Findings of the Association for Computational Linguistics: EMNLP 2026
Spatial audio editing modifies an existing soundfield according to a user’s instruction while preserving the rest of the scene. Unlike conventional audio editing, it must reason jointly about audio events, spatial information, dynamic changes, and environmental information in first-order Ambisonic (FOA) waveforms. Existing language-guided editors mainly target conventional audio or rely on sequential operations, and therefore do not directly support one-stage editing for complex 3D spatial instructions. We present SwanWeave, the first one-stage multi-task framework for instruction-guided 3D FOA spatial audio editing. We build paired FOA supervision from open-source speech and sound-effect corpora using controllable room simulation, covering more than ten single-operation and compound tasks across the four editing axes. To handle this heterogeneous edit space, SwanWeave uses Spatial Edit Mixture-of-Experts (SE-MoE) with dual-level routing, selecting task-aware expert combinations for compound instructions and frame-level routed/null experts for local edit decisions. We further introduce Spatial Preference Optimization (SPO), a Direct Preference Optimization (DPO)-based alignment objective with edit-specific negative targets, and adopt staged training to improve natural-language grounding. Experiments show that SwanWeave achieves better editing quality than existing general audio editors and spatial audio baselines across all tasks. Spatial audio editing demos can be found at https://swanaigc.github.io/#swanweave, code can be found at: https://github.com/MM-Speech/SwanWeave.
DynSess: Dynamic Session-Level Evaluation and Optimization Framework for Role-Playing Agents
Rongsheng Zhang | Jiji Tang | Junnan Ren | Zuyi Bao | Weijie Chen | Ruofan Hu | Tangjie Lv | Zhou Zhao | Yan Zhang
Findings of the Association for Computational Linguistics: EMNLP 2026
Rongsheng Zhang | Jiji Tang | Junnan Ren | Zuyi Bao | Weijie Chen | Ruofan Hu | Tangjie Lv | Zhou Zhao | Yan Zhang
Findings of the Association for Computational Linguistics: EMNLP 2026
Role-playing with large language models is fundamentally a *session-level* task, requiring agents to sustain character identity and interaction quality across extended multi-turn conversations. Yet existing evaluation and optimization methods remain largely turn-level, failing to capture long-horizon quality. We propose **DynSess**, a unified session-level framework for role-playing agents. **DynSess-Eval** scores complete dialogue sessions via rubrics targeting long-horizon behaviors. Leveraging its session-level rewards, we construct high-quality training trajectories through multi-turn lookahead search and train **DynSess-Character** with two complementary variants: DSPO (off-policy) and GSRPO (on-policy). Experiments show that DynSess-Eval aligns with human judgments substantially better than prior evaluators, and blind human evaluation further shows that DynSess-Character matches the strongest character model despite using substantially fewer parameters, while maintaining strong role consistency and interactive ability. The dataset and code are publicly available in https://github.com/Collab-Gen/DynSess.
From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
Rongsheng Zhang | Ruofan Hu | Weijie Chen | Jiji Tang | Junnan Ren | Wanying Wu | Xunuoyan Chen | Tangjie Lv | Tao Jin | Zhou Zhao
Findings of the Association for Computational Linguistics: EMNLP 2026
Rongsheng Zhang | Ruofan Hu | Weijie Chen | Jiji Tang | Junnan Ren | Wanying Wu | Xunuoyan Chen | Tangjie Lv | Tao Jin | Zhou Zhao
Findings of the Association for Computational Linguistics: EMNLP 2026
While role-playing agents excel in short-term interactions, long-term conversations overwhelm context windows, motivating external memory frameworks. Current systems typically rely on persona-agnostic summarization, which records facts without persona-specific interpretation, yielding generic responses that compromise persona fidelity. To bridge this gap, we introduce RoleMemo, a dataset featuring four reasoning tasks where the factual fragments must be interpreted through the persona to reach the correct answer. Evaluation on RoleMemo exposes critical limitations of persona-agnostic frameworks. We thus propose DualMem, which decouples memory into two streams: factual cognition and persona-conditioned insight. Trained through Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL), our framework with a 4B-parameter model outperforms zero-shot persona-agnostic frameworks powered by DeepSeek-V3.2 for sustained persona fidelity. Our resources are available at https://github.com/Collab-Gen/RoleMemo.
Doc-REFRAG: Rethinking Multimodal Document Retrieval-Augmented Generation
Ruofan Hu | Shengyang Xu | Minjie Hong | Xiaoda Yang | Sashuai Zhou | Ke Lei | Tao Jin | Zhou Zhao
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Ruofan Hu | Shengyang Xu | Minjie Hong | Xiaoda Yang | Sashuai Zhou | Ke Lei | Tao Jin | Zhou Zhao
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Real-world knowledge resides in multimodal documents, necessitating retrieval-augmented generation (RAG) for accurate question answering. However, existing multimodal RAG models are primarily designed for single-image or closed-document settings and exhibit limited accuracy in realistic multi-image scenarios. Moreover, processing numerous retrieved images incurs substantial computational overhead from irrelevant visual tokens. To address these challenges, we introduce DocLongRAG, a large-scale dataset of 343K question–answer pairs, each associated with an average of 37.4 retrieved images to reflect authentic RAG workflows. Building on this dataset, we propose Doc-REFRAG, a question-guided framework that compresses visual tokens into coarse chunks and selectively expands question-relevant ones via a lightweight RL-based selector. Experiments on six benchmarks show that Doc-REFRAG outperforms eleven strong baselines, achieving state-of-the-art accuracy with significantly lower inference latency. Our resources are available at https://github.com/Collab-Gen/Doc-REFRAG.
2024
Prompt-Singer: Controllable Singing-Voice-Synthesis with Natural Language Prompt
Yongqi Wang | Ruofan Hu | Rongjie Huang | Zhiqing Hong | Ruiqi Li | Wenrui Liu | Fuming You | Tao Jin | Zhou Zhao
Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)
Yongqi Wang | Ruofan Hu | Rongjie Huang | Zhiqing Hong | Ruiqi Li | Wenrui Liu | Fuming You | Tao Jin | Zhou Zhao
Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)
Recent singing-voice-synthesis (SVS) methods have achieved remarkable audio quality and naturalness, yet they lack the capability to control the style attributes of the synthesized singing explicitly. We propose Prompt-Singer, the first SVS method that enables attribute controlling on singer gender, vocal range and volume with natural language. We adopt a model architecture based on a decoder-only transformer with a multi-scale hierarchy, and design a range-melody decoupled pitch representation that enables text-conditioned vocal range control while keeping melodic accuracy. Furthermore, we explore various experiment settings, including different types of text representations, text encoder fine-tuning, and introducing speech data to alleviate data scarcity, aiming to facilitate further research. Experiments show that our model achieves favorable controlling ability and audio quality. Audio samples are available at http://prompt-singer.github.io .
2022
TWEET-FID: An Annotated Dataset for Multiple Foodborne Illness Detection Tasks
Ruofan Hu | Dongyu Zhang | Dandan Tao | Thomas Hartvigsen | Hao Feng | Elke Rundensteiner
Proceedings of the Thirteenth Language Resources and Evaluation Conference
Ruofan Hu | Dongyu Zhang | Dandan Tao | Thomas Hartvigsen | Hao Feng | Elke Rundensteiner
Proceedings of the Thirteenth Language Resources and Evaluation Conference
Foodborne illness is a serious but preventable public health problem – with delays in detecting the associated outbreaks resulting in productivity loss, expensive recalls, public safety hazards, and even loss of life. While social media is a promising source for identifying unreported foodborne illnesses, there is a dearth of labeled datasets for developing effective outbreak detection models. To accelerate the development of machine learning-based models for foodborne outbreak detection, we thus present TWEET-FID (TWEET-Foodborne Illness Detection), the first publicly available annotated dataset for multiple foodborne illness incident detection tasks. TWEET-FID collected from Twitter is annotated with three facets: tweet class, entity type, and slot type, with labels produced by experts as well as by crowdsource workers. We introduce several domain tasks leveraging these three facets: text relevance classification (TRC), entity mention detection (EMD), and slot filling (SF). We describe the end-to-end methodology for dataset design, creation, and labeling for supporting model development for these tasks. A comprehensive set of results for these tasks leveraging state-of-the-art single-and multi-task deep learning methods on the TWEET-FID dataset are provided. This dataset opens opportunities for future research in foodborne outbreak detection.
Search
Fix author
Co-authors
- Zhou Zhao 5
- Tao Jin 3
- Weijie Chen 2
- Ke Lei 2
- Ruiqi Li 2
- Tangjie Lv 2
- Junnan Ren 2
- Jiji Tang 2
- Rongsheng Zhang 2
- Sashuai Zhou 2
- Zuyi Bao 1
- Xunuoyan Chen 1
- Hao Feng 1
- Wenxiang Guo 1
- Thomas Hartvigsen 1
- Minjie Hong 1
- Zhiqing Hong 1
- Rongjie Huang 1
- Yongshi Li 1
- Wenrui Liu 1
- Changhao Pan 1
- Elke Rundensteiner 1
- Dandan Tao 1
- Yongqi Wang 1
- Chenyuhao Wen 1
- Wanying Wu 1
- Haorui Xu 1
- Shengyang Xu 1
- Xiaoda Yang 1
- Xiang Yin 1
- Fuming You 1
- Dongyu Zhang 1
- Yan Zhang 1
- Yu Zhang 1