With the growing availability of open-source task-specific models, model merging has become a practical way to combine multiple models into one. However, existing studies mainly focus on fully fine-tuned models and overlook widely used LoRA modules. We show that full fine-tuning merging methods perform poorly on LoRA due to its larger cross-task magnitude disparities, which distort merged parameters and degrade performance. To address this, we propose DO-Merging (Decouple and Orthogonalize), a data-free framework for LoRA merging. Specifically, we decompose model weights into magnitude and direction components and merge them independently, preventing magnitude imbalances from distorting the merged direction. We further introduce a layer-wise gradient descent method with orthogonal constraints that reduces inter-task interference among direction components without requiring any training data. We provide theoretical analysis supporting both components. Extensive experiments across vision, language, and multi-modal domains show that DO-Merging consistently outperforms existing methods, achieving up to 5% gains on vision tasks, with negligible additional cost. Both parts can be integrated into existing merging methods as plug-and-play enhancements.