Low-Rank Adaptation (LoRA) is widely used for fine-tuning large language models, but suffers from catastrophic forgetting. Existing continual learning methods mitigate this by estimating per-parameter importance independently. However, this assumption breaks down under LoRA’s factorization 𝛥 W = BA: infinitely many pairs (B, A) yield the same weight update, making importance scores on A and B separately unreliable. We propose HLoRA, which computes path-integral importance in the combined weight update space, bypassing the unreliable separate estimation on A and B. HLoRA applies hierarchical regularization at two levels: element-wise constraints preserve parameters critical for general knowledge, and layer-wise coefficients adapt regularization strength based on each layer’s aggregate importance. Experiments on scientific, medical, and physical tasks using GPT-J and LLaMA-3 show that HLoRA mitigates catastrophic forgetting while enhancing adaptability, running approximately 20x faster and requiring only 10%–15% of the storage compared to prior methods. Code is released on https://anonymous.4open.science/r/HLoRA-915F.

Small Language Models (SLMs) have been widely deployed in edge computing and mobile devices. Generally, these models need to be continuously updated to adapt to dynamic environments and evolving user needs. Therefore, the performance of SLMs in continual learning has become a critical research focus. Compared to LLMs with rich redundant representation spaces, SLMs are more likely to overwrite previous representations during new task learning due to their fewer attention heads and layers. To address this issue, we propose LightLoRA, a sparse fine-tuning method specifically designed for SLMs. This method constructs a conflict-aware importance mechanism by combining raw importance from SVD decomposition with task conflict degrees based on gradient direction consistency. By selecting and updating heads that provide high contributions and low conflicts, LightLoRA successfully maintains sufficient space for knowledge preservation. Experimental results show that the proposed method reaches state-of-the-art performance across multiple benchmarks without adding extra parameters. Code is released on https://anonymous.4open.science/r/LightLoRA-077C.