Wuxiaobin
Author directory2026
Mitigating Object Hallucinations in Vision-Language Models via Head-Level Attribution and Counteractive Pruning
Wu Liyan | Xin Luo | Fan Wu | Yingen Liu | Wuxiaobin
Findings of the Association for Computational Linguistics: EMNLP 2026
Wu Liyan | Xin Luo | Fan Wu | Yingen Liu | Wuxiaobin
Findings of the Association for Computational Linguistics: EMNLP 2026
Large vision-language models (LVLMs) excel at multimodal tasks but often generate instance-level object hallucinations, describing nonexistent objects. Yet most mitigation strategies rely on output-level or coarse-grained interventions, limiting their ability to identify and selectively regulate attention heads associated with faithful versus hallucinatory generation, often at the cost of descriptive coverage. To address this limitation, we propose HACP, a training-free framework that integrates head-level semantic attribution with instance-adaptive counteractive pruning. HACP introduces InfoSpectralScore to summarize spectral changes in head-perturbed representations and identify heads associated with hallucination or grounded generation, which are then selectively suppressed or reinforced during decoding. Across multiple LVLMs and benchmarks, HACP consistently reduces hallucinations while preserving descriptive coverage; on a 1000-image multi-label COCO subset, it lowers CHAIR-s/i from 66.7/18.0 to 43.4/8.6 and improves F1 from 73.0 to 77.0. These findings suggest that head-level attribution provides an actionable signal for fine-grained inference-time hallucination control in LVLMs, while helping preserve descriptive coverage. The official project repository for HACP is available at https://github.com/S1yang/HACP.