TIA-VL: Diagnosing and Correcting Template-Induced Ambiguity in Medical Vision-Language Decisions

Yifan Luo, Jiaran Shi, Guoming Wang, Siliang Tang


Abstract
Medical vision-language models (VLMs) are often evaluated by report retrieval, grounding, and explanation quality. We show that these metrics can create a metric illusion: a model may appear clinically grounded while exploiting acquisition template, crop, orientation, or protocol rather than patient anatomy. We formalize this failure as template-induced ambiguity and introduce TIA-VL, a counterfactual benchmark with source, template-family, severity, and compound protocol shifts. We also propose Rationale-Grounded Template Ambiguity Calibration (R-GTAC), a plug-in calibration layer that discounts template-dominated evidence while preserving anatomy and textual rationale support. Across strong medical VLM-style baselines, including BioViL-T-style frozen features, adapter features, and anatomy-teacher features, R-GTAC reduces wrong same-template errors to 0.0000 on the main protocol splits. We then test whether systems know when to abstain through readiness and counterfactual rationale deferral. Counterfactual rationale readiness filtering further identifies a reliable low-risk 25% subset reaching 0.8500 Top-1 on the hardest compound shift, while revealing that coupled protocol shifts pose critical frontiers for global risk calibration. Thus, TIA-VL supports a conservative message for medical VLM evaluation: models should be judged not only by whether they answer, but by whether their apparent rationales survive anti-shortcut counterfactuals. The code and benchmark will be available at https://github.com/YifanLuo0103/TIA-VL.
Anthology ID:
2026.findings-emnlp.538
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2026
Month:
October
Year:
2026
Address:
Budapest, Hungary
Editors:
Hung-Yi Lee, Sunipa Dev, Jessy Li, Desmond Elliott
Venue:
Findings
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
11303–11319
Language:
URL:
https://aclanthology.org/2026.findings-emnlp.538/
DOI:
Bibkey:
Cite (ACL):
Yifan Luo, Jiaran Shi, Guoming Wang, and Siliang Tang. 2026. TIA-VL: Diagnosing and Correcting Template-Induced Ambiguity in Medical Vision-Language Decisions. In Findings of the Association for Computational Linguistics: EMNLP 2026, pages 11303–11319, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):
TIA-VL: Diagnosing and Correcting Template-Induced Ambiguity in Medical Vision-Language Decisions (Luo et al., Findings 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.findings-emnlp.538.pdf