In this paper, we introduce layer-wise curriculum learning aimed at efficient LLM compression. The proposed method facilitates the transfer of knowledge from the teacher to student model, utilizing a curriculum learning approach that begins with easier optimization tasks and progressively tackles much harder ones. In order to adopt the layer-wise learning in LLM compression, we partition the whole model into multiple segments consisting of layers, thereby enabling more computationally efficient knowledge transfer of LLMs. Based on the theoretical evidence of cumulative error phenomenon, layer-wise curriculum learning accelerates convergence while stabilizing the knowledge transfer process. In addition, we present a feature caching method with a multi-threading strategy to efficiently address feature misalignment across layers, maximizing GPU utilization. Consequently, our method exhibits advanced model compression performance, as well as high computational efficiency in terms of minimized memory usage and short training hours. Experiments on multiple datasets show that the proposed method achieves state-of-the-art performance while reducing GPU memory usage and training hours by more than 50% on BERT and GPT-2. Moreover, it outperforms the other pruning methods on LLaMA-family and Qwen under the same training hours, with a lower GPU memory footprint. Upon acceptance, we will publicly release the code to support the reproducibility of our results.