Zijie Geng
Author directory2026
PAPO: Stabilizing Rubric Integration Training via Decoupled Advantage Normalization
Zelin Tan | Zhouliang Yu | Bohan Lin | Zijie Geng | Hejia Geng | Yudong Zhang | Mulei Zhang | Yang Chen | Shuyue Hu | Zhenfei Yin | Chen Zhang | Lei Bai
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Zelin Tan | Zhouliang Yu | Bohan Lin | Zijie Geng | Hejia Geng | Yudong Zhang | Mulei Zhang | Yang Chen | Shuyue Hu | Zhenfei Yin | Chen Zhang | Lei Bai
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
We propose Process-Aware Policy Optimization (PAPO), a method that integrates process-level evaluation into Group Relative Policy Optimization (GRPO) through **decoupled advantage normalization**, to address two limitations of existing reward designs. Outcome reward models (ORM) evaluate only final-answer correctness, treating all correct responses identically, and gradually lose the advantage signal as groups become uniformly correct. Rubric-based process reward models (PRM) offer richer supervision, differentiating quality among correct responses and restoring advantage diversity, but using their scores as rewards triggers verbosity-driven reward hacking and accuracy collapse. PAPO resolves both by composing the advantage from an outcome advantage Aout, derived from outcome reward rout over all responses, and a process advantage Aproc, derived from process reward rproc exclusively over the correct subset. Aout anchors correctness, and correct subset normalization on Aproc rules out verbosity-driven hacking while sustaining the gradient on all-correct groups where ORM exhausts. PAPO consistently outperforms ORM across five model configurations and six benchmarks (reaching 51.3% vs. 46.3% on OlympiadBench for Qwen2.5-7B while continuing to improve as ORM plateaus and declines), with analogous gains when composed with mainstream GRPO variants.