Jiafu Tang
Author directory2026
OmniHalluc-L: Counterfactual Benchmarking and Modality-Perturbation Calibration for Long-Form Omni Hallucination
Zixuan Dong | Jiafu Tang | Zhide Lei | Zhe Cao | Zijie Zhang | Yanghai Wang | Shihao Li | Xiaodong Wang | Baoyun Peng | Jiaheng Liu
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Zixuan Dong | Jiafu Tang | Zhide Lei | Zhe Cao | Zijie Zhang | Yanghai Wang | Shihao Li | Xiaodong Wang | Baoyun Peng | Jiaheng Liu
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Long-video Omni assistants often fail not by inventing content, but by misbinding real evidence: they hear the right utterance and see the right event, yet attach it to the wrong speaker, moment, or modality. These almost-true errors evade standard video QA because local evidence remains valid, so item-level scoring can reward both a supported claim and its near-counterfactual. We introduce a counterfactual event-binding protocol that constructs paired supported/counterfactual claims from the same audio-visual event evidence and evaluates them by strict-pair accuracy. We instantiate it as OmniHalluc-L, to our knowledge the first benchmark for long-video Omni hallucination, with 3600 single-claim QA items from 638 long-form videos averaging 24.16 minutes and covering 256.87 hours. Under this protocol, open-weight Omni models remain weak at pair-level binding: Qwen2.5-Omni-7B reaches 32.06% and Qwen3-Omni-Instruct reaches 41.55%, versus 76.54% for a closed-source reference. To narrow this gap, we propose MPRC (Modality-Perturbation Reliability Calibration), a frozen-backbone calibrator that combines native confidence with response shifts from selected audio-negative probes. MPRC raises these two models to 36.22% and 51.09%, respectively, and improves target-adapted MCQ accuracy on OmniVideoBench and WorldSense. The code is available at https://github.com/ZexDong/omnihallucL-mprc.
CapRiCorn-1K: A Comprehensive Benchmark for Video Captioning and Subject Referential Consistency Across Temporal Scales
Xinlong Chen | Jiafu Tang | Yue Ding | Yizhuo Jia | Bozhou Li | Bohan Zeng | Yang Shi | Shihao Li | Yiyan Ji | Qiang Liu | Weihong Lin | Yuanxing Zhang | Pengfei Wan | Liang Wang | Tieniu Tan
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Xinlong Chen | Jiafu Tang | Yue Ding | Yizhuo Jia | Bozhou Li | Bohan Zeng | Yang Shi | Shihao Li | Yiyan Ji | Qiang Liu | Weihong Lin | Yuanxing Zhang | Pengfei Wan | Liang Wang | Tieniu Tan
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Accurate and comprehensive video captions with consistent subject references are critical for downstream understanding and generation tasks. However, few existing benchmarks can objectively and comprehensively evaluate these properties across diverse durations and scenarios, thereby hindering the advancement of video captioning models. To bridge this gap, we propose CapRiCorn-1K, a comprehensive benchmark designed to evaluate both video captioning quality and subject referential consistency across long temporal horizons and diverse video domains. To accommodate varied evaluation needs, our benchmark supports both audiovisual and visual-only settings. Extensive experiments on CapRiCorn-1K reveal that current models generally struggle to generate accurate and comprehensive captions while maintaining consistent subject references. Moreover, as video duration increases, both the overall caption quality and subject referential consistency decline. Notably, our evaluation metrics exhibit strong correlations with the performance of downstream understanding and generation tasks conditioned on the generated captions, further validating their effectiveness. CapRiCorn-1K will be fully open-sourced to facilitate future research.
AVSCap: Orchestrating Audio-Visual Synergy for Omni-modal Video Captioning
Yanghai Wang | Jiahao Wang | Jiafu Tang | Yuanxing Zhang | Zhe Cao | Hanyan Bian | Zijie Zhang | Weiliang Luo | Zhiyu Pan | Zixuan Dong | Jiaheng Liu | Zhaoxiang Zhang
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Yanghai Wang | Jiahao Wang | Jiafu Tang | Yuanxing Zhang | Zhe Cao | Hanyan Bian | Zijie Zhang | Weiliang Luo | Zhiyu Pan | Zixuan Dong | Jiaheng Liu | Zhaoxiang Zhang
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Omni-modal video captioning is not merely combining visual captioning with audio transcription: a useful caption must describe how visual actions, speech, music, and sound effects co-evolve over time. Existing large multimodal models often fail at this relational step, treating audio and visual streams as loosely coupled observations, relying heavily on automatic speech recognition, and under-specifying non-speech sounds and their links to visual events. We present AVSCap, a framework for audio-visual captioning centered on explicit cross-modal event binding. First, we construct AVSCap-130K, a tri-modal training corpus generated by a decoupled-then-fused pipeline that anchors visual and acoustic evidence independently before composing temporally grounded omni-modal captions. Second, we train AVSCap-7B, a 7B captioner with a two-stage strategy: supervised fine-tuning establishes baseline captioning ability, while a sample-efficient reinforcement learning stage uses hybrid rewards to optimize acoustic completeness and audio-visual synergy. Our scaling analysis shows that reinforcement learning brings larger gains than simply increasing SFT data. Third, we introduce AVSCapBench, a human-curated benchmark that decomposes captions into visual, audio, and synergy events and evaluates them with fine-grained event recall. Experiments on AVSCapBench and external benchmarks show that AVSCap-7B improves non-speech audio coverage and cross-modal binding, delivering the best overall performance among evaluated open-source models.