Fine-tuning Vision-Language Models (VLMs) is a common practice to adapt pre-trained models for downstream tasks. Yet existing fine-tuning methods consistently keep the pre-trained architecture intact, leaving open the question of whether maintaining such structural integrity is truly necessary for effective adaptation. To investigate this, we propose Divide-and-Adapt, a new fine-tuning paradigm that departs from preserving the full pre-trained structure. It first divides the pre-trained model into multiple downstream-oriented subnetworks, and then adapts them collaboratively through a routing-based coordination mechanism. This design facilitates the selection of structures that are most beneficial for downstream tasks and enables their collaborative integration. Remarkably, even without any task-specific training, our Divide-and-Adapt framework first divides the pre-trained model into downstream-oriented subnetworks and adaptively activates them for downstream inference, already outperforming the pre-trained model on downstream tasks. With further fine-tuning, Divide-and-Adapt surpasses both full fine-tuning and parameter-efficient baselines across multiple benchmarks, demonstrating its effectiveness. Our code will be made publicly available at https://github.com/haiiling/DivideAndAdapt.