Guangjia Chai

Author directory

2026

Process Reward Models (PRMs) are vulnerable to reward hacking: policies exploit surface cues such as confident phrasing or structured formatting in place of logical correctness, which yields an inverted-U collapse under optimization pressure. We propose ARMOR-PRM, which augments standard PRM training with three regularizers: a counterfactual margin loss, an adversarial invariance loss with low-score anchoring, and a gradient penalty on the reward head. On ProcessBench (three splits; "math" excluded due to training data overlap), ARMOR-PRM improves Detection Rate by 14.9 percentage points and AUROC by 4.1 percentage points over Vanilla PRM, and is the only variant whose Best-of-N reranking accuracy improves monotonically through N=128. A domain-shift diagnostic further shows that objective-level robustness and data coverage are distinct but complementary axes: ARMOR-PRM responds best to iterative hard-negative retraining, suggesting a practical two-stage deployment path.