DiaVLo: Diagnosing Behaviours of Vision-Language Models

Lorenzo Corti, Jie Yang


Abstract
Vision-language models (VLMs) rely on storing and transferring appropriate information across their sub-components. Verifying that the VLMs exhibit desired behaviours, while avoiding harmful ones, is central to their reliable deployment. Yet, methods that identify VLM behaviours remain scarce. We present DiaVLo, a diagnostic framework that leverages human curation and VLMs’ generation capabilities to construct specifications of desired and observed VLM behaviours, surfacing potential misalignments. Beyond this, DiaVLo also provides causal estimates to identify the most influential concepts steering VLM behaviours. We evaluate DiaVLo on several open-source VLMs under both classification and generation conditions. Our experiments show that DiaVLo produces behaviour labels that correlate with model performance and provide context for measured performance. DiaVLo surfaced behaviours that are clearly aligned and misaligned, alongside patterns in how VLMs perceive, organise, and prioritise concepts.
Anthology ID:
2026.findings-emnlp.1841
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2026
Month:
October
Year:
2026
Address:
Budapest, Hungary
Editors:
Hung-Yi Lee, Sunipa Dev, Jessy Li, Desmond Elliott
Venue:
Findings
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
39299–39332
Language:
URL:
https://aclanthology.org/2026.findings-emnlp.1841/
DOI:
10.18653/2026.findings-emnlp.1841
Bibkey:
Cite (ACL):
Lorenzo Corti and Jie Yang. 2026. DiaVLo: Diagnosing Behaviours of Vision-Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2026, pages 39299–39332, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):
DiaVLo: Diagnosing Behaviours of Vision-Language Models (Corti & Yang, Findings 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.findings-emnlp.1841.pdf