Reliability Challenges in Diffusion Vision–Language Models

Md. Atabuzzaman, Chris Thomas


Abstract
Diffusion-based Large Vision-Language Models (dLVLMs) have recently emerged as a compelling alternative to autoregressive (AR) LVLMs, offering advantages in parallel decoding, bidirectional context, and controllable generation. Despite rapid progress, their reliability properties remain largely uncharacterized. We present the first systematic reliability evaluation of hallucination and bias in dLVLMs, benchmarking six diffusion models against competitive AR baselines across four dimensions. Our key findings are: (1) dLVLMs reverse the yes-bias of AR models in binary visual queries; (2) they achieve competitive hallucination rates yet exhibit degraded linguistic quality; (3) they collapse to near-zero accuracy on underrepresented racial groups with opposite-polarity gender bias; and (4) they exhibit accuracy collapse in multiple-choice settings when the correct option is shorter than its distractors, associated with a length prior that emerges at the first denoising step. Tokens committed at late denoising steps with low confidence further correlate with hallucinated content, pointing to a mechanistic signal unique to diffusion generation. These patterns vary across model families, suggesting reliability is shaped by the generative paradigm together with training data.
Anthology ID:
2026.emnlp-main.2561
Volume:
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Month:
October
Year:
2026
Address:
Budapest, Hungary
Editors:
Hung-Yi Lee, Sunipa Dev, Jessy Li, Desmond Elliott
Venue:
EMNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
58201–58216
Language:
URL:
https://aclanthology.org/2026.emnlp-main.2561/
DOI:
Bibkey:
Cite (ACL):
Md. Atabuzzaman and Chris Thomas. 2026. Reliability Challenges in Diffusion Vision–Language Models. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 58201–58216, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):
Reliability Challenges in Diffusion Vision–Language Models (Atabuzzaman & Thomas, EMNLP 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.emnlp-main.2561.pdf