Shamanthak Hegde
Author directory2026
Where Hallucinations Live: A Cross-Architecture Circuit in VQ-Tokenized Vision-Language Models
Shamanthak Hegde | Xiangrui Liu | Maitreya Patel | YZ Yang
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Shamanthak Hegde | Xiangrui Liu | Maitreya Patel | YZ Yang
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Unified vision-language models (VLMs) that tokenize images through a vector-quantized (VQ) codebook routinely hallucinate objects on grounded yes/no benchmarks, yet existing decoding-time fixes treat this as generic miscalibration without an architectural account. Using activation patching across twenty-five models spanning eight LLM families, we identify an early-layer (L₀) attention routing circuit shared across VQ-tokenized VLMs and propose a three-gate diagnostic that distinguishes the models carrying it. The diagnostic isolates ten positive models (five natural unified-VQ VLMs across three LLM families and five induced variants) and rejects the remaining fifteen. A single-variable architectural swap (LLaVA-1.6 CLIP+MLP → VQ+Linear) installs the circuit, while a matched-compute MLP control does not, isolating vector quantization as the source of the pathological signal; the routing pathway that carries it is one that the backbone already provides. Against tuned baselines, DoLA wins on binary calibration, but only L₀ ablation reduces object hallucination in open-ended generation (CHAIRᵢ -31% relative; tuned DoLA and VCD leave it unchanged or worsen it). These results recast object hallucination in unified VQ VLMs as a property of architecture and pretraining, and yield a targeted intervention that mechanism-agnostic decoding cannot replicate.