Ranjit Raut
Author directory2026
WINNAR749 at SHROOM-Visions: Model-Agnostic Fine-Grained Hallucination Detection in VLMs via QLoRA Approach with Soft, Calibrated Targets
Ranjit Raut
The Proceedings of the Third Workshop on Uncertainty-Aware NLP
Ranjit Raut
The Proceedings of the Third Workshop on Uncertainty-Aware NLP
Vision-Language Models (VLMs) are showing remarkable performance in connecting vision and language, closely intertwined. However, they are vulnerable to hallucination and alignment problems. To address these issues, this paper describes a simple system for the SHROOM-visions shared task on character-level hallucination detection in image-conditioned text. The system attaches two lightweight linear heads to a pretrained VLM backbone and fine-tunes them, together with low-rank adapters on the backbone. As a result, they predict a per-character hallucination probability and a per-character distribution over five hallucination classes. We train directly against soft targets as the released annotations are themselves soft (overlapping, disagreeing multi-annotator spans), rather than collapsing them into hard span labels and fit a post-hoc isotonic calibration map so that predicted probabilities track empirical annotator agreement, the quantity the task’s Confidence Calibration metric rewards. The system we submitted to the official shared-task evaluation built on Qwen2.5-VL-3B-Instruct contained two implementation issues that we identified only after the submission deadline; we report the official results as submitted, document the issues and their fixes, and separately report unofficial post-evaluation results obtained after correcting them and porting the pipeline to Qwen3.5-4B.[The code is available at <https://github.com/theranjitraut/SHROOM-Visions-2026>]