Weiliang Luo
Author directory2026
AVSCap: Orchestrating Audio-Visual Synergy for Omni-modal Video Captioning
Yanghai Wang | Jiahao Wang | Jiafu Tang | Yuanxing Zhang | Zhe Cao | Hanyan Bian | Zijie Zhang | Weiliang Luo | Zhiyu Pan | Zixuan Dong | Jiaheng Liu | Zhaoxiang Zhang
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Yanghai Wang | Jiahao Wang | Jiafu Tang | Yuanxing Zhang | Zhe Cao | Hanyan Bian | Zijie Zhang | Weiliang Luo | Zhiyu Pan | Zixuan Dong | Jiaheng Liu | Zhaoxiang Zhang
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Omni-modal video captioning is not merely combining visual captioning with audio transcription: a useful caption must describe how visual actions, speech, music, and sound effects co-evolve over time. Existing large multimodal models often fail at this relational step, treating audio and visual streams as loosely coupled observations, relying heavily on automatic speech recognition, and under-specifying non-speech sounds and their links to visual events. We present AVSCap, a framework for audio-visual captioning centered on explicit cross-modal event binding. First, we construct AVSCap-130K, a tri-modal training corpus generated by a decoupled-then-fused pipeline that anchors visual and acoustic evidence independently before composing temporally grounded omni-modal captions. Second, we train AVSCap-7B, a 7B captioner with a two-stage strategy: supervised fine-tuning establishes baseline captioning ability, while a sample-efficient reinforcement learning stage uses hybrid rewards to optimize acoustic completeness and audio-visual synergy. Our scaling analysis shows that reinforcement learning brings larger gains than simply increasing SFT data. Third, we introduce AVSCapBench, a human-curated benchmark that decomposes captions into visual, audio, and synergy events and evaluates them with fine-grained event recall. Experiments on AVSCapBench and external benchmarks show that AVSCap-7B improves non-speech audio coverage and cross-modal binding, delivering the best overall performance among evaluated open-source models.