Many continual learning methods for LLMs mitigate forgetting by expanding task-specific modules, yet this leads to memory costs that grow with the number of tasks. We reveal that such expansion may be unnecessary: a single LoRA adapter possesses substantial per-task parameter redundancy, where retaining only 3–10% of LoRA-B entries suffices to recover over 95% of task accuracy. Moreover, the per-parameter update magnitudes follow a heavy-tailed distribution whose geometric knee point naturally separates critical parameters from dispensable ones. Building on this insight, we propose Continual Adaptation via Subspace Trimming (CAST), which adaptively allocates a compact task-specific subspace within a shared LoRA adapter using the knee point criterion, freezes previously assigned parameters to eliminate cross-task interference, and routes inputs to relevant subspaces via learned semantic vectors without requiring task identities. Experiments on LLaMA-3.1-8B and Qwen3-8B demonstrate that CAST matches per-task LoRA accuracy across 15 sequential tasks while maintaining a fixed adapter budget with over 24% idle capacity preserved.

Continual pre-training (CPT) has been an important approach for adapting language models to specific domains or tasks. In this paper, we comprehensively study its key designs to balance the new abilities while retaining the original abilities, and present an effective CPT method that can greatly improve the Chinese language ability and scientific reasoning ability of LLMs. To achieve it, we design specific data mixture and curriculum strategies based on existing datasets and synthetic high-quality data. Concretely, we synthesize multidisciplinary scientific QA pairs based on related web pages to guarantee the data quality, and also devise the performance tracking and data mixture adjustment strategy to ensure the training stability. For the detailed designs, we conduct preliminary studies on a relatively small model, and summarize the findings to help optimize our CPT method. Extensive experiments on a number of evaluation benchmarks show that our approach can largely improve the performance of Llama-3 (8B), including both the general abilities (+8.81 on C-Eval and +6.31 on CMMLU) and the scientific reasoning abilities (+12.00 on MATH and +4.13 on SciEval). Our model, data, and codes are available at https://github.com/RUC-GSAI/Llama-3-SynE.