Jaegwang Shin

Author directory

2026

Decoding-time mitigation of vision-language model (VLM) hallucination applies one fixed intervention and typically trades recall for lower hallucination. We trace this trade-off to treating hallucination as a single failure mode. Measuring the normalised entropy H of leave-one-out attributions over panoptic image segments, we find that a VLM’s inference mass concentrates in one task-conditional tail of H: captioning in the over-concentration tail (low H, 53–66%), binary question-answering in the over-spread tail (high H, 74–95%). The pattern holds in 6/6 (backbone × benchmark) cells on LLaVA-1.5-7B and Qwen2.5-VL-7B, survives exact Shapley recomputation, reproduces within AMBER, and is shared by three further backbones. Because errors and correct outputs share the tail—H is a regime signal, not a hallucination detector—an intervention matched to the wrong tail actively hurts: blank-image contrastive decoding raises captioning CHAIR_S from 49.3 to 60.3. The regime can be read without attribution at inference, from blank-image agreement and on-image token margin; SBC (Segment-Based Conditioning) uses them to route between segment boosting and probabilistic marginalisation. With one shared configuration, SBC lowers CHAIR_I while preserving recall and raises POPE F1 on both backbones (+1.95 / +3.31), at 1.7–1.9× greedy cost on captioning and 4.9× on single-token QA.