Zheng Yang

Author directory

2026

Integrating supervised fine-tuning (SFT) and reinforcement learning (RL) has become a common post-training paradigm for improving the reasoning capabilities of large language models. Existing methods typically rely on sequential SFT–RL pipelines or static loss weights, overlooking sample-level heterogeneity and evolving learning states. We propose Sample Learning State (SLS), a rollout-based characterization of each training sample using two complementary signals: sample mastery and exploration-trajectory dispersion. These signals estimate whether reward-driven updates are reliable for the core task and whether current rollouts provide informative contrast for RL optimization, respectively. Based on SLS, we design a state-aware coefficient that dynamically balances SFT and RL losses, increasing the RL weight only when the model demonstrates sufficient core-task competence and the sampled trajectories provide useful reward contrast. Experiments on Qwen2.5-Math models show that SLS achieves state-of-the-art results on four in-distribution and two out-of-distribution mathematical reasoning benchmarks. In multi-reward settings, SLS preserves core reasoning performance while mitigating reward hacking induced by auxiliary rewards.
Long chain-of-thought (CoT) reasoning in large language models (LLMs) improves performance on complex tasks but incurs substantial inference cost, although many queries do not require extended computation. Existing approaches typically reduce this cost via uniform length penalties or fixed computation budgets, but lack length constraints aligned with the model’s own query-dependent distribution of successful reasoning lengths. We propose a reinforcement learning framework that aligns the generated reasoning length with query-dependent successful computation scales estimated from verified trajectories. By providing adaptive guidance derived from the distribution of successful reasoning lengths, our method avoids the limitations of global length regularization. Experiments on mathematical reasoning benchmarks demonstrate improved accuracy–efficiency trade-offs, reducing unnecessary reasoning while maintaining or improving task performance.