MedProb: Probing Internal Representations of Vision-Language Models for Medical Question Answering

Erfan Nourbakhsh, Ke Yang, Anthony Rios


Abstract
Medical visual question answering (Med-VQA) is often assumed to require medical fine-tuning, large models, or complex multi-agent pipelines. We revisit this assumption with MedProb, a lightweight probing framework that predicts multiple-choice Med-VQA answers from frozen VLM representations without free-text generation. Across PATH-VQA, SLAKE, and VQA-RAD, MedProb recovers substantially more answer-relevant signal than prompting and performs stronger than medical VLMs and agentic systems. Probing also reduces the apparent gap between small and large models compared to prompting, suggesting that smaller VLMs contain more recoverable Med-VQA signal than generation-based evaluation reveals. Across 14 matched general-purpose and medical VLM pairs, medical adaptation does not consistently improve this linear decodability.
Anthology ID:
2026.findings-emnlp.1320
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2026
Month:
October
Year:
2026
Address:
Budapest, Hungary
Editors:
Hung-Yi Lee, Sunipa Dev, Jessy Li, Desmond Elliott
Venue:
Findings
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
27961–27999
Language:
URL:
https://aclanthology.org/2026.findings-emnlp.1320/
DOI:
10.18653/2026.findings-emnlp.1320
Bibkey:
Cite (ACL):
Erfan Nourbakhsh, Ke Yang, and Anthony Rios. 2026. MedProb: Probing Internal Representations of Vision-Language Models for Medical Question Answering. In Findings of the Association for Computational Linguistics: EMNLP 2026, pages 27961–27999, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):
MedProb: Probing Internal Representations of Vision-Language Models for Medical Question Answering (Nourbakhsh et al., Findings 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.findings-emnlp.1320.pdf