VisER: Visual Evidence and Reliance for Object Hallucination Detection in LVLMs

Afsaneh Hasanebrahimi, Hanxun Huang, Christopher Leckie, Sarah Erfani


Abstract
Object hallucination remains a persistent reliability issue in large vision-language models, where generated object mentions may sound plausible but lack visual grounding. Recent training-free detectors use internal signals such as token likelihood, attention, visual confidence, or image-text similarity to identify hallucinated objects. These signals are useful, but they are often source-confounded. They measure how strongly an object is supported inside the model without distinguishing whether that support comes from object-specific visual evidence or the generated text prefix. In difficult cases, a hallucinated object can still receive high internal support because it fits the scene, is associated with nearby visual cues, or follows naturally from the generated text prefix. We propose VisER, a training-free two-sided metric for object-level hallucination detection. VisER evaluates each generated object mention from two complementary views. Visual Evidence measures whether object-context compatibility is backed by object-specific evidence from image tokens. Visual Reliance measures whether the object is supported more by the image than by the generated prefix. Combining these views gives a more source-aware grounding score, while avoiding additional object-level verification generations. Across multiple LVLMs and benchmarks, VisER improves AUROC and AUPR over a range of baselines.
Anthology ID:
2026.emnlp-main.2645
Volume:
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Month:
October
Year:
2026
Address:
Budapest, Hungary
Editors:
Hung-Yi Lee, Sunipa Dev, Jessy Li, Desmond Elliott
Venue:
EMNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
60186–60202
Language:
URL:
https://aclanthology.org/2026.emnlp-main.2645/
DOI:
Bibkey:
Cite (ACL):
Afsaneh Hasanebrahimi, Hanxun Huang, Christopher Leckie, and Sarah Erfani. 2026. VisER: Visual Evidence and Reliance for Object Hallucination Detection in LVLMs. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 60186–60202, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):
VisER: Visual Evidence and Reliance for Object Hallucination Detection in LVLMs (Hasanebrahimi et al., EMNLP 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.emnlp-main.2645.pdf