Quan Fang
Author directory2026
CoDe-LoRA: Mitigating the Orthogonality Dilemma in Continual Learning of LLMs via Knowledge Consolidation and Decoupling
Maoqi Liu | Quan Fang | Yufei He
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Maoqi Liu | Quan Fang | Yufei He
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Continual learning (CL) is essential for Large Language Models (LLMs) to sequentially adapt to evolving tasks. To mitigate catastrophic forgetting, recent advances implement low-rank adaptation with orthogonal projections (e.g., O-LoRA) to isolate task parameters. However, we reveal that such strict geometric constraints trigger an “Orthogonality Dilemma”: rigid parameter isolation impedes the transfer and accumulation of shared representations across semantically related tasks. In this work, we propose a new replay-free method, called ̲Consolidation and ̲Decoupling ̲LoRA (CoDe-LoRA), for CL of LLMs. CoDe-LoRA disentangles the learning process into Consolidating\ Universal\ Knowledge and Decoupling\ Task-Specific\ Knowledge. To achieve this, CoDe-LoRA leverages an adaptive null-space projection mechanism and semantic routing to balance knowledge accumulation with task-specific adaptation. Experimental results on diverse CL benchmarks demonstrate that CoDe-LoRA consistently outperforms state-of-the-art methods. Our code is available at [https://github.com/Estrellajer/CoDe-LoRA](https://github.com/Estrellajer/CoDe-LoRA).