Shinhyoung Jang

Author directory

2026

Supervised fine-tuning is the standard approach for adapting pretrained large language models to specialized domains, yet it treats the pretrained checkpoint as a fixed and neutral initialization. In practice, pretrained models encode broad general-domain priors that can conflict with target-domain learning. We propose Forget-to-Focus (F2F), a two-stage domain-specialization framework that first applies controlled unlearning to reshape the pretrained checkpoint and then performs ordinary supervised fine-tuning from the resulting initialization. A key challenge is deciding what to forget. We introduce NG-DS, a neuron-guided data selection method that identifies pretraining-dominant and domain-dominant feed-forward neurons, then selects forget examples that strongly activate the former while weakly activating the latter. This model-aware forget set directs unlearning toward pretraining-like behavior while reducing the risk of erasing target-domain-relevant knowledge.We evaluate F2F on five LLM backbones spanning 0.6B-72B parameters across coding, medical question answering, and mathematical reasoning. F2F consistently improves over standard SFT and adaptation baselines. Across coding, medical QA, and mathematical reasoning, F2F consistently improves over standard tuning. On HumanEval, it improves over SFT by +13.84 for Qwen3-0.6B, +7.86 for LLaMA3.1-8B, and +10.14 for LLaMA2-13B. On PubMedQA, it improves Qwen3-0.6B from 62.60 to 80.50, showing that targeted preparatory forgetting provides a better initialization for tuning.