Yin Xie
Author directory2026
Consistency-Calibrated Advantage Estimation for Label-Free Reinforcement Learning
Meng Yang | Yongle Zhao | Zhiyang Zhang | Xiaorui Jiang | Peirou Liang | Furui Chen | Yin Xie | Ziyong Feng
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Meng Yang | Yongle Zhao | Zhiyang Zhang | Xiaorui Jiang | Peirou Liang | Furui Chen | Yin Xie | Ziyong Feng
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Reinforcement learning from unlabeled prompts offers a promising path to improving LLM reasoning without curated supervision or verifiable rewards. However, existing label-free methods typically rely on either intrinsic self-certainty or extrinsic answer agreement, both of which can become unreliable in hard reasoning regimes: confident trajectories can be wrong, while majority answers can drift toward stable yet incorrect attractors. We propose CCAE, a verifier-free and parameter-free advantage estimator that formulates label-free GRPO as a multi-signal advantage estimation problem. CCAE decomposes the advantage into a base self-certainty term, an intrinsic evidence factor, and an extrinsic agreement factor, with their interaction adaptively calibrated by a group-level proxy for agreement reliability, without introducing any reward model, verifier, or trainable parameters. On MATH-Train, CCAE improves over the strongest unlabeled baseline by 5.0 and 4.3 points on Qwen3-4B/8B-Base, respectively, matching supervised GRPO. On the harder DAPO-14k-MATH, CCAE remains stable while prior unlabeled methods collapse, outperforming their best pre-collapse checkpoints by up to 11.1 points. Results on Natural Reasoning further suggest that CCAE remains effective beyond math-only training. Code is available at: [https://github.com/tifing/CCAE](https://github.com/tifing/CCAE).