Ozkan Kilic
2026
Fine-Tuning Small Language Models for Cybersecurity: Data Ordering, Knowledge Distillation, and the Educator Effect
Ozkan Kilic | Raja Soundaramourty | Ramu Chenchaiah
Proceedings of the Second International Conference on Natural Language Processing and Artificial Intelligence for Cyber Security
Ozkan Kilic | Raja Soundaramourty | Ramu Chenchaiah
Proceedings of the Second International Conference on Natural Language Processing and Artificial Intelligence for Cyber Security
Data-sovereignty rules forbid cloud-hosted AI in many high-security environments, leaving compact on-premise models as the only path to AI-assisted cybersecurity. We fine-tune three small open-source models, Gemma 2 2B, Phi-3.5 3.8B, Llama 3.1 8B, on ~147,600 synthetic cybersecurity QA pairs using QLoRA on V100 GPUs. Under strict MCQ evaluation Gemma 2 gains +9.3 pp, Phi +4.0 pp, and Llama drops −24.0 pp. We term this the educator effect: models trained on pedagogical data internalize explanatory behavior at the expense of format compliance. Severity appears to scale with capacity, though capacity is confounded with architecture and learning rate. A controlled ablation shows randomized ordering outperforms curriculum, without significance testing on the 75-question exam.