Jingyi Tang
Author directory2026
Exploring Hallucination of Large Multimodal Models in Video Understanding: Benchmark, Analysis and Mitigation
Hongcheng Gao | Jiashu Qu | Jingyi Tang | Baolong Bi | Yue Liu | Hongyu Chen | Liang Li | Li Su | Qingming Huang
Transactions of the Association for Computational Linguistics, Volume 14
Hongcheng Gao | Jiashu Qu | Jingyi Tang | Baolong Bi | Yue Liu | Hongyu Chen | Liang Li | Li Su | Qingming Huang
Transactions of the Association for Computational Linguistics, Volume 14
Hallucinations in large multimodal models (LMMs), where responses appear correct but are factually wrong, limit their reliability in practical applications. This paper focuses on hallucinations in video understanding, a more dynamic and challenging domain than static modalities. We present HAVEN, a comprehensive benchmark with 6K questions spanning three dimensions: hallucination causes, aspects, and question formats. Then, we systematically evaluate 18 LMMs and analyze seven influential factors affecting hallucinations, including video duration, model size, and reasoning capabilities. Additionally, we explore two reasoning-based strategies to reduce hallucinations in LMMs: (1) text-based reasoning via rejection fine-tuning on reasoning datasets and segment-weighted Direct Preference Optimization, and (2) video-based reasoning through a tool-augmented frame-level operations on video optimized via supervised tool learning and Hallucination-aware Group Relative Policy Optimization (HGRPO). Extensive experiments and analyses demonstrate the effectiveness. Remarkably, video-based reasoning strategy improves the Qwen2.5-VL-7B-Instruct by 10.16% in accuracy and reduces the bias score by 4.89%.