Retrieval-Augmented Generation (RAG) systems are increasingly adopted in medical question answering, where generated answers depend heavily on retrieved biomedical evidence. However, demographic perturbations in clinical queries can substantially alter retrieval behavior, causing retrievers to surface demographically correlated yet clinically irrelevant evidence. We refer to this phenomenon as retrieval shift, where demographic-sensitive retrieval representations propagate bias into downstream generation. To investigate this issue, we construct a demographically perturbed MedQA dataset and systematically analyze retrieval instability under demographic perturbations across representative medical RAG systems. Our analysis shows that RAG systems are substantially more sensitive to demographic perturbations than standalone LLMs, indicating that retrieval plays a central role in propagating demographic bias. To mitigate retrieval shift, we propose Bias-Aware Neutral Debiasing (BAND), a rotation-based retrieval debiasing framework that suppresses demographic-sensitive retrieval directions while maintaining clinically relevant retrieval behavior. Experiments demonstrate that our method consistently reduces retrieval shift while largely preserving downstream task performance across multiple medical RAG systems. Our code and dataset are available at: https://github.com/hrimlee/BAND