Juntong Zhou

Author directory

2026

Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced the reasoning capabilities of Large Language Models (LLMs), while being prone to entropy collapse, a rapid degradation of policy entropy that forces the model into premature convergence and limits its performance ceiling. However, existing solutions for mitigating entropy collapse often struggle to reconcile the trade-off between exploration capability and convergence efficiency. To address this, we propose Variance-aware Negative Feedback Policy Optimization (VFPO), a framework that dynamically regulates the policy entropy through the lens of group statistics. We first mathematically prove that policy entropy is intrinsically coupled with the variance and mean of group probabilities for sampled trajectories. Leveraging this insight, VFPO operates via a dual-phase strategy: an initial exploration phase employs a closed-loop negative feedback mechanism via both sample-level and token-level reward modulation to maintain group variance and prevent the mean trajectory probability from saturating; then followed by an annealing phase that relaxes these constraints to facilitate deterministic convergence. Extensive experiments on models of distinct scales (1.5B, 3B, 7B) demonstrate that VFPO yields average improvements of 3.34%, 6.52%, and 6.03%, respectively, over the DAPO baseline, and achieves state-of-the-art performance across diverse mathematical benchmarks, effectively expanding the exploration boundary while achieving superior reasoning accuracy. To facilitate reproducibility, we release the code, data and trained models at the link https://github.com/carpe1/VFPO.