Dading Chong
Author directory2026
VFPO: Variance-aware Negative Feedback Policy Optimization for Mitigating Entropy Collapse via Group Statistics
Bo Zhang | Zinan Zeng | Juntong Zhou | Dading Chong | Zhenyu Ding | Dengpeng Wu | Ning Ding | Peilin Zhou
Findings of the Association for Computational Linguistics: EMNLP 2026
Bo Zhang | Zinan Zeng | Juntong Zhou | Dading Chong | Zhenyu Ding | Dengpeng Wu | Ning Ding | Peilin Zhou
Findings of the Association for Computational Linguistics: EMNLP 2026
Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced the reasoning capabilities of Large Language Models (LLMs), while being prone to entropy collapse, a rapid degradation of policy entropy that forces the model into premature convergence and limits its performance ceiling. However, existing solutions for mitigating entropy collapse often struggle to reconcile the trade-off between exploration capability and convergence efficiency. To address this, we propose Variance-aware Negative Feedback Policy Optimization (VFPO), a framework that dynamically regulates the policy entropy through the lens of group statistics. We first mathematically prove that policy entropy is intrinsically coupled with the variance and mean of group probabilities for sampled trajectories. Leveraging this insight, VFPO operates via a dual-phase strategy: an initial exploration phase employs a closed-loop negative feedback mechanism via both sample-level and token-level reward modulation to maintain group variance and prevent the mean trajectory probability from saturating; then followed by an annealing phase that relaxes these constraints to facilitate deterministic convergence. Extensive experiments on models of distinct scales (1.5B, 3B, 7B) demonstrate that VFPO yields average improvements of 3.34%, 6.52%, and 6.03%, respectively, over the DAPO baseline, and achieves state-of-the-art performance across diverse mathematical benchmarks, effectively expanding the exploration boundary while achieving superior reasoning accuracy. To facilitate reproducibility, we release the code, data and trained models at the link https://github.com/carpe1/VFPO.
2024
TS-Align: A Teacher-Student Collaborative Framework for Scalable Iterative Finetuning of Large Language Models
Chen Zhang | Chengguang Tang | Dading Chong | Ke Shi | Guohua Tang | Feng Jiang | Haizhou Li
Findings of the Association for Computational Linguistics: EMNLP 2024
Chen Zhang | Chengguang Tang | Dading Chong | Ke Shi | Guohua Tang | Feng Jiang | Haizhou Li
Findings of the Association for Computational Linguistics: EMNLP 2024
Mainstream approaches to aligning large language models (LLMs) heavily rely on human preference data, particularly when models require periodic updates. The standard process for iterative alignment of LLMs involves collecting new human feedback for each update. However, the data collection process is costly and challenging to scale. To address this issue, we introduce the “TS-Align” framework, which fine-tunes a policy model using pairwise feedback data automatically mined from its outputs. This automatic mining process is efficiently accomplished through the collaboration between a large-scale teacher model and a small-scale student model. The policy fine-tuning process can be iteratively repeated using on-policy generations within our proposed teacher-student collaborative framework. Through extensive experiments, we demonstrate that our final aligned policy outperforms the base policy model with an average win rate of 69.7% across seven conversational or instruction-following datasets. Furthermore, we show that the ranking capability of the teacher is effectively distilled into the student through our pipeline, resulting in a small-scale yet effective reward model for policy model alignment.
Search