By continuously acquiring new knowledge, continual learning enables large language models (LLMs) to remain adaptive, accurate, and relevant as the world evolves. In continual learning, tasks arrive one after another and the model learns them sequentially. As more tasks are introduced, existing methods for learning a new task impose increasing computational demands. However, we find that the computational overhead, which gradually increases as the number of tasks grows, can be mitigated. To limit the continual increase in computational costs, we introduce a simple but efficient block-sparse (EBS) framework for continual learning of LLMs. In EBS, we employ a dedicated, task-specific low-rank adaptation (LoRA) block for each task. When training on a new task, only one LoRA block is loaded into GPU memory and involved in computation, while at inference each input activates at most two LoRA blocks. This design ensures that the computational cost remains independent of the number of tasks, allowing EBS to maintain high efficiency even when the sequence of tasks becomes very large. We offer a simple implementation that works across different model sizes and transformer architectures (decoder-only and encoder-decoder) while consistently delivering both high efficiency and strong task performance. Experimental results on two continual learning benchmarks demonstrate that EBS achieves state-of-the-art performance on the T5 and LLaMA 2 models.

Despite recent progress in systematic evaluation frameworks, benchmarking the uncertainty of large language models (LLMs) remains a highly challenging task. Existing methods for benchmarking the uncertainty of LLMs face three key challenges: the need for internal model access, additional training, or high computational costs. This is particularly unfavorable for closed-source models. To this end, we introduce UBench, a new benchmark for evaluating the uncertainty of LLMs. Unlike other benchmarks, UBench is based on confidence intervals. It encompasses 11,978 multiple-choice questions spanning knowledge, language, understanding, and reasoning capabilities. Based on this, we conduct extensive experiments. This includes comparisons with other advanced uncertainty estimation methods, the assessment of the uncertainty of 20 LLMs, and an exploration of the effects of Chain-of-Thought (CoT) prompts, role-playing (RP) prompts, and temperature on model uncertainty. Our analysis reveals several crucial insights: 1) Our confidence interval-based methods are highly effective for uncertainty quantification; 2) Regarding uncertainty, outstanding open-source models show competitive performance versus closed-source models; 3) CoT and RP prompts present potential ways to improve model reliability, while the influence of temperature changes follows no universal rule. Our implementation is available at https://github.com/Cyno2232/UBENCH.