Zijie Geng

Author directory

2026

We propose Process-Aware Policy Optimization (PAPO), a method that integrates process-level evaluation into Group Relative Policy Optimization (GRPO) through **decoupled advantage normalization**, to address two limitations of existing reward designs. Outcome reward models (ORM) evaluate only final-answer correctness, treating all correct responses identically, and gradually lose the advantage signal as groups become uniformly correct. Rubric-based process reward models (PRM) offer richer supervision, differentiating quality among correct responses and restoring advantage diversity, but using their scores as rewards triggers verbosity-driven reward hacking and accuracy collapse. PAPO resolves both by composing the advantage from an outcome advantage Aout, derived from outcome reward rout over all responses, and a process advantage Aproc, derived from process reward rproc exclusively over the correct subset. Aout anchors correctness, and correct subset normalization on Aproc rules out verbosity-driven hacking while sustaining the gradient on all-correct groups where ORM exhausts. PAPO consistently outperforms ORM across five model configurations and six benchmarks (reaching 51.3% vs. 46.3% on OlympiadBench for Qwen2.5-7B while continuing to improve as ORM plateaus and declines), with analogous gains when composed with mainstream GRPO variants.