Xiaonan Luo
Author directory2026
Right Answers for the Right Reasons: Coherence-Enhanced Policy Optimization for Large Language Models
Xiangqi Wang | Yue Huang | Yujun Zhou | Xiaonan Luo | Kehan Guo | Yuchen Ma | Xiangliang Zhang
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Xiangqi Wang | Yue Huang | Yujun Zhou | Xiaonan Luo | Kehan Guo | Yuchen Ma | Xiangliang Zhang
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Large language models (LLMs) trained with reinforcement objectives often achieve superficially correct answers via shortcut strategies, pairing correct outputs with spurious or unfaithful reasoning and degrading under small input perturbations. We introduce Coherence-Enhanced Policy Optimization (CE-PO), a model agnostic reward fusion framework that augments policy optimization with a differentiable Jacobian coherence signal along the generation pathway from prompt (X) to rationale (Y) to answer (Z). CE-PO estimates internal gradient influence through Jacobian-based sensitivities, uses a reshuffling-based residualization procedure to reduce shortcut-driven reward hacking, and fuses the resulting coherence score with task-accuracy via an Lp combiner that controls the strength of coherence regularization. The unified reward integrates with PPO/GRPO without architectural changes. Across multiple reasoning benchmarks, CE-PO improves accuracy over baselines by 5.49 on average (up to 9.58), while increasing the model’s faithful reasoning capability (12.14 % on average across five backbones).