Recent parameter-efficient fine-tuning (PEFT) methods effectively adapt Vision-Language Models (VLMs) to seen tasks, yet often exhibit limited generalization to unseen novel classes. To better understand this phenomenon, we analyze adaptation dynamics from the perspective of cross-modal mutual information (MI). Our empirical analyses reveal a consistent pattern across existing methods: during the adaptation process, cross-modal MI for novel classes frequently encounters early plateaus or declines, indicating weakened cross-modal semantic dependency. Motivated by this observation, we propose Retrieval-Augmented Dynamic Adaptation (RADA), a retrieval-enhanced adaptation framework that continuously injects rich instance-specific semantic priors through an LLM-enhanced dual-modality memory bank and a lightweight modality-agnostic token generator. By facilitating fine-grained cross-modal alignment while preserving pretrained zero-shot ability, RADA achieves more transferable adaptation across diverse scenarios. Extensive experiments across multiple widely used benchmarks demonstrate that RADA consistently achieves state-of-the-art performance while maintaining strong zero-shot capability. In addition, our empirical analyses show that RADA preserves more stable cross-modal dependency trajectories throughout adaptation, offering further insights into the relationship between adaptation dynamics and generalization in VLMs.