Caihong Cheng
Author directory2026
APEX : Adaptively Balancing Off-Policy Guidance and On-Policy Exploration for LLM Reasoning
Yuchen Wang | Zikang Zhang | Duo Xu | Kunpeng Xu | Qihui Zhu | Wei Huang | Siying Wu | Caihong Cheng | Xin He | Yang Yang | Guanjun Jiang | Xiaoyan Sun
Findings of the Association for Computational Linguistics: EMNLP 2026
Yuchen Wang | Zikang Zhang | Duo Xu | Kunpeng Xu | Qihui Zhu | Wei Huang | Siying Wu | Caihong Cheng | Xin He | Yang Yang | Guanjun Jiang | Xiaoyan Sun
Findings of the Association for Computational Linguistics: EMNLP 2026
In post-training large language models (LLMs), supervised fine-tuning (SFT) provides expert guidance but can overfit, while reinforcement learning (RL) enables exploration but often suffers from cold-start and sparse-reward issues. Although combining SFT with RL is intuitive, naively mixing off-policy expert data into on-policy RL can induce distribution shift and destabilize optimization. We propose APEX, a framework that harmonizes off-policy guidance and on-policy exploration via a Cognitive State Awareness mechanism. Specifically, APEX leverages a perplexity-based Student’s -test to compare the sequence-level perplexity distributions of on-policy generations against candidate offline data. This comparison diagnoses whether failures reflect sampling stochasticity or missing knowledge. Guided by this diagnosis, APEX adaptively gates GRPO optimization and modulates SFT supervision. The framework preferentially selects offline samples that are statistically consistent with the current on-policy distribution to guide RL updates, while filtering out high-shift data and verifiably incorrect samples that could destabilize training. APEX also uses a dynamically weighted SFT loss when needed. Extensive experiments show that APEX improves over SFT-then-RL by +7.10 points in average pass@1 on six math benchmarks and by +15.50 points on three general reasoning benchmarks.