Large vision-language models (LVLMs) often hallucinate due to over-reliance on language priors. Existing methods implicitly assume that visual tokens maintain consistent semantics across layers and thus indiscriminately inject shallow-layer visual information into deeper layers. We identify a new phenomenon, termed correlation flip, where a visual token positively correlated with the decoding context in an intermediate layer becomes negatively correlated in the final layer, indicating that correct visual evidence is suppressed before prediction. Based on this finding, we propose Correlation-Flip Visual Injection (CFVI), a simple and training-free decoding-time method that selectively identifies correlation-flip tokens and injects corrective visual signals into the final-layer hidden state to restore suppressed evidence and mitigate hallucinations. CFVI can be directly applied to existing LVLMs, and introduces minimal computational overhead. Experiments on multiple benchmarks demonstrate that CFVI achieves strong performance while preserving language fluency.

Fine-tuned Large Language Models (LLMs) often demonstrate poor calibration, with their confidence scores misaligned with actual performance. While calibration has been extensively studied in models trained from scratch, the impact of LLMs’ prior knowledge on calibration during fine-tuning remains understudied. Our research reveals that LLMs’ prior knowledge causes potential poor calibration due to the ubiquitous presence of known data in real-world fine-tuning, which appears harmful for calibration. Specifically, data aligned with LLMs’ prior knowledge would induce overconfidence, while new knowledge improves calibration. Our findings expose a tension: LLMs’ encyclopedic knowledge, while enabling task versatility, undermines calibration through unavoidable knowledge overlaps. To address this, we propose CogCalib, a cognition-aware framework that applies targeted learning strategies according to the model’s prior knowledge. Experiments across 7 tasks using 3 LLM families prove that CogCalib significantly improves calibration while maintaining performance, achieving an average 57% reduction in ECE compared to standard fine-tuning in Llama3-8B. These improvements generalize well to out-of-domain tasks, enhancing the objectivity and reliability of domain-specific LLMs, and making them more trustworthy for critical human-AI interaction applications.