Large Vision-Language Models (LVLMs) are increasingly explored for multimodal medical diagnosis, their ability to conduct accurate consultation remains challenging because of unstable visual reasoning and limited tracking of patient information across dialogue turns. To address this problem we present V-RAM (Visual Retrieval-Augmented Multi-Agent Reasoning Framework), a training-free framework for improving LVLM-based multimodal medical dialogue diagnosis. V-RAM improves medical dialogue diagnosis by adding visual retrieval, structured memory, and multi-agent reasoning around a pretrained LVLM. It retrieves similar medical images as diagnostic cues and tracks patient evidence across dialogue turns. The framework operates at inference time, it requires no supervised fine-tuning and can be used with different LVLM backbones. Experiments on benchmark show that V-RAM outperforms existing baselines and the current LVLMs by 7.3% on an average with 86% lesser number of turns that are used for medical dialogue diagnosis.