Recent parameter-efficient fine-tuning (PEFT) methods effectively adapt Vision-Language Models (VLMs) to seen tasks, yet often exhibit limited generalization to unseen novel classes. To better understand this phenomenon, we analyze adaptation dynamics from the perspective of cross-modal mutual information (MI). Our empirical analyses reveal a consistent pattern across existing methods: during the adaptation process, cross-modal MI for novel classes frequently encounters early plateaus or declines, indicating weakened cross-modal semantic dependency. Motivated by this observation, we propose Retrieval-Augmented Dynamic Adaptation (RADA), a retrieval-enhanced adaptation framework that continuously injects rich instance-specific semantic priors through an LLM-enhanced dual-modality memory bank and a lightweight modality-agnostic token generator. By facilitating fine-grained cross-modal alignment while preserving pretrained zero-shot ability, RADA achieves more transferable adaptation across diverse scenarios. Extensive experiments across multiple widely used benchmarks demonstrate that RADA consistently achieves state-of-the-art performance while maintaining strong zero-shot capability. In addition, our empirical analyses show that RADA preserves more stable cross-modal dependency trajectories throughout adaptation, offering further insights into the relationship between adaptation dynamics and generalization in VLMs.

As the scale of large language models (LLMs) grows and natural language tasks become increasingly diverse, Parameter-Efficient Fine-Tuning (PEFT) has become the standard paradigm for fine-tuning LLMs. Among PEFT methods, LoRA is widely adopted for not introducing additional inference overhead. However, existing LoRA’s shared parameter space paradigm introduces parameter interference, leading to a gap in generalization performance for specific tasks compared to full fine-tuning. To address this issue, we propose a parameter-separated low-rank adapter, called Subspace Low-Rank Adaptation (SuLoRA). The core idea of SuLoRA is to account for task differences by decomposing LoRA’s parameter matrix into multiple independent subspaces and assigning them differentially to distinct tasks. This prevents interference across tasks and enhances the effectiveness of low-rank adaptation. Additionally, SuLoRA achieves higher rank expansion by freezing the A matrix, further improving generalization capability. We conduct extensive experiments on various NLP tasks, demonstrating that SuLoRA significantly outperforms LoRA in trainable parameter efficiency and overall model performance. Furthermore, we validate SuLoRA’s effectiveness in domain generalization and multi-modal tasks, showcasing its strong generalization ability.