Pengfei Wan
Author directory2026
CapRiCorn-1K: A Comprehensive Benchmark for Video Captioning and Subject Referential Consistency Across Temporal Scales
Xinlong Chen | Jiafu Tang | Yue Ding | Yizhuo Jia | Bozhou Li | Bohan Zeng | Yang Shi | Shihao Li | Yiyan Ji | Qiang Liu | Weihong Lin | Yuanxing Zhang | Pengfei Wan | Liang Wang | Tieniu Tan
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Xinlong Chen | Jiafu Tang | Yue Ding | Yizhuo Jia | Bozhou Li | Bohan Zeng | Yang Shi | Shihao Li | Yiyan Ji | Qiang Liu | Weihong Lin | Yuanxing Zhang | Pengfei Wan | Liang Wang | Tieniu Tan
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Accurate and comprehensive video captions with consistent subject references are critical for downstream understanding and generation tasks. However, few existing benchmarks can objectively and comprehensively evaluate these properties across diverse durations and scenarios, thereby hindering the advancement of video captioning models. To bridge this gap, we propose CapRiCorn-1K, a comprehensive benchmark designed to evaluate both video captioning quality and subject referential consistency across long temporal horizons and diverse video domains. To accommodate varied evaluation needs, our benchmark supports both audiovisual and visual-only settings. Extensive experiments on CapRiCorn-1K reveal that current models generally struggle to generate accurate and comprehensive captions while maintaining consistent subject references. Moreover, as video duration increases, both the overall caption quality and subject referential consistency decline. Notably, our evaluation metrics exhibit strong correlations with the performance of downstream understanding and generation tasks conditioned on the generated captions, further validating their effectiveness. CapRiCorn-1K will be fully open-sourced to facilitate future research.
Closing the Verification Loop: Self-Check Captioning for Long-Paragraph Detailed Audio Captioning
Fengji Ma | Yan Rong | Xu Li | Chen Zhang | Pengfei Wan | Li Liu
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Fengji Ma | Yan Rong | Xu Li | Chen Zhang | Pengfei Wan | Li Liu
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Long-paragraph detailed audio captioning, which requires dense and transcript-faithful descriptions of fine-grained audio content, remains unsolved for current audio-visual multimodal language models. We attribute this failure to two structural problems. The first is data poverty, as no public corpus jointly provides long clips, paragraph captions, and verbatim-transcript fidelity. The second is generation-mode failure, evidenced by a 44.8 to 46.4 percentage-point gap between right-audio and shuffled-audio multiple-choice question (MCQ) accuracy. We address both within Self-Check Captioning (SCC), a unified framework that instantiates audio-grounded question answering as the verification primitive at every lifecycle stage. SCC yields three artifacts. Long-paragraph Audio Caption 50k (LACap-50k) is a 50,222-clip audio-visual corpus with 491.5-word captions and a post-hoc automatic speech recognition (ASR) audit. Layer-Curvature Supervised Fine-Tuning (LC-SFT) is the first on-policy supervised fine-tuning method to weight tokens by intermediate-layer evidence, motivated by our identification of Late-Layer Semantic-Entropy Collapse (SEC). SCC-Verifier arbitrates among caption rollouts via audio-grounded self-answering at inference. Across multiple benchmarks, our system attains state-of-the-art among open-source captioners and is competitive with proprietary baselines. We release LACap-50k to fill the resource gap for long-paragraph detailed audio captioning research.
SegTune: Structured and Fine-Grained Control for Song Generation
Yuejiao Wang | Zihao Ji | Pengfei Cai | Xu Li | Haorui Zheng | Zewen Song | Zhongliang Liu | Chen Zhang | Pengfei Wan
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Yuejiao Wang | Zihao Ji | Pengfei Cai | Xu Li | Haorui Zheng | Zewen Song | Zhongliang Liu | Chen Zhang | Pengfei Wan
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Recent advances in neural song generation have enabled high-quality synthesis from lyrics and global textual prompts. However, most systems fail to model temporally varying attributes of songs, severely limiting fine-grained control over musical structure and dynamics. To address this, we propose Segtune, a Diffusion Transformer-based framework enabling structured and fine-grained controllability by allowing users or large language models (LLMs) to specify local musical descriptions aligned to song segments. These segment prompts are temporally broadcast to corresponding time windows, while global prompts ensure stylistic coherence. To support precise lyric-to-music alignment, we introduce an LLM-based duration predictor that autoregressively generates sentence-level timestamps in LyRiCs format. We further construct a large-scale data pipeline for high-quality song collection with aligned lyrics and prompts, and propose new metrics to evaluate segment alignment and vocal consistency. Experiments demonstrate that Segtune outperforms existing baselines in both musicality and controllability. Visit our demo page for codes and more generated songs.
2025
SEA: Supervised Embedding Alignment for Token-Level Visual-Textual Integration in MLLMs
Yuanyang Yin | Yaqi Zhao | Yajie Zhang | Yuanxing Zhang | Ke Lin | Jiahao Wang | Xin Tao | Pengfei Wan | Wentao Zhang | Feng Zhao
Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing
Yuanyang Yin | Yaqi Zhao | Yajie Zhang | Yuanxing Zhang | Ke Lin | Jiahao Wang | Xin Tao | Pengfei Wan | Wentao Zhang | Feng Zhao
Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing
Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities by integrating visual and textual inputs, yet modality alignment remains one of the most challenging aspects. Current MLLMs typically rely on simple adapter architectures and pretraining approaches to bridge vision encoders with large language models (LLM), guided by image-level supervision. We identify this paradigm often leads to suboptimal alignment between modalities, significantly constraining the LLM’s ability to properly interpret and reason with visual features particularly for smaller language models. To address this fundamental limitation, we propose Supervised Embedding Alignment (SEA), a token-level supervision alignment method that enables more precise visual-text alignment during pretraining. SEA introduces minimal computational overhead while preserving language capabilities and substantially improving cross-modal understanding. Our comprehensive analyses reveal critical insights into the adapter’s role in multimodal integration, and extensive experiments demonstrate that SEA consistently improves performance across various model sizes, with smaller models benefiting the most (average performance gain of 7.61% for Gemma-2B). This work establishes a foundation for developing more effective alignment strategies for future multimodal systems.
RICO: Improving Accuracy and Completeness in Image Recaptioning via Visual Reconstruction
Yuchi Wang | Yishuo Cai | Shuhuai Ren | Sihan Yang | Linli Yao | Yuanxin Liu | Yuanxing Zhang | Pengfei Wan | Xu Sun
Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing
Yuchi Wang | Yishuo Cai | Shuhuai Ren | Sihan Yang | Linli Yao | Yuanxin Liu | Yuanxing Zhang | Pengfei Wan | Xu Sun
Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing
Image recaptioning is widely used to generate training datasets with enhanced quality for various multimodal tasks. Existing recaptioning methods typically rely on powerful multimodal large language models (MLLMs) to enhance textual descriptions, but often suffer from inaccuracies due to hallucinations and incompleteness caused by missing fine-grained details. To address these limitations, we propose RICO, a novel framework that refines captions through visual reconstruction. Specifically, we leverage a text-to-image model to reconstruct a caption into a reference image, and prompt an MLLM to identify discrepancies between the original and reconstructed images to refine the caption. This process is performed iteratively, further progressively promoting the generation of more faithful and comprehensive descriptions. To mitigate the additional computational cost induced by the iterative process, we introduce RICO-Flash, which learns to generate captions like RICO using DPO. Extensive experiments demonstrate that our approach significantly improves caption accuracy and completeness, outperforms most baselines by approximately 10% on both CapsBench and CompreCap.
Search
Co-authors
- Yuanxing Zhang 3
- Xu Li 2
- Chen Zhang 2
- Pengfei Cai 1
- Yishuo Cai 1
- Xinlong Chen 1
- Yue Ding 1
- Yiyan Ji 1
- Zihao Ji 1
- Yizhuo Jia 1
- Bozhou Li 1
- Shihao Li 1
- Ke Lin 1
- Weihong Lin 1
- Li Liu 1
- Qiang Liu 1
- Yuanxin Liu 1
- Zhongliang Liu 1
- Fengji Ma 1
- Shuhuai Ren 1
- Yan Rong 1
- Yang Shi 1
- Zewen Song 1
- Xu Sun 1
- Tieniu Tan 1
- Jiafu Tang 1
- Xin Tao 1
- Jiahao Wang 1
- Liang Wang 1
- Yuchi Wang 1
- Yuejiao Wang 1
- Sihan Yang 1
- Linli Yao 1
- Yuanyang Yin 1
- Bohan Zeng 1
- Wentao Zhang 1
- Yajie Zhang 1
- Feng Zhao 1
- Yaqi Zhao 1
- Haorui Zheng 1