While large language models (LLMs) show immense potential in specialized domains, their adaptation is often hindered by limited access to domain-specific data and the high computational cost of model training. To address these challenges, we propose a lightweight knowledge injection framework. Instead of sharing raw data directly, our approach fine-tunes a small model and transfers its knowledge vectors to the target LLM for knowledge injection, thereby reducing data exposure and computational overhead. To mitigate the knowledge loss incurred during the injection process, we further introduce an iterative knowledge calibration strategy. We theoretically bound the error of the transformation matrix and prove the stability and linear convergence of the calibration process. Comprehensive experiments demonstrate that our method significantly outperforms parameter-efficient baselines and achieves performance comparable to full-parameter fine-tuning, while consuming only 28% of the GPU memory and 3% of the raw training data. Code is available at https://github.com/Yuhanleeee/LiSKI.