Kunpeng Xu

Author directory

2026

In post-training large language models (LLMs), supervised fine-tuning (SFT) provides expert guidance but can overfit, while reinforcement learning (RL) enables exploration but often suffers from cold-start and sparse-reward issues. Although combining SFT with RL is intuitive, naively mixing off-policy expert data into on-policy RL can induce distribution shift and destabilize optimization. We propose APEX, a framework that harmonizes off-policy guidance and on-policy exploration via a Cognitive State Awareness mechanism. Specifically, APEX leverages a perplexity-based Student’s -test to compare the sequence-level perplexity distributions of on-policy generations against candidate offline data. This comparison diagnoses whether failures reflect sampling stochasticity or missing knowledge. Guided by this diagnosis, APEX adaptively gates GRPO optimization and modulates SFT supervision. The framework preferentially selects offline samples that are statistically consistent with the current on-policy distribution to guide RL updates, while filtering out high-shift data and verifiably incorrect samples that could destabilize training. APEX also uses a dynamically weighted SFT loss when needed. Extensive experiments show that APEX improves over SFT-then-RL by +7.10 points in average pass@1 on six math benchmarks and by +15.50 points on three general reasoning benchmarks.