@inproceedings{zhao-etal-2026-process,
title = "Process Rewarding via Solution-Space Alignment",
author = "Zhao, ChenZhuo and
Huang, Yue and
Zhao, Pu and
Yang, Fangkai and
Wang, Lu and
Ma, Lin and
Yang, Zhi and
Huang, Youling and
Wang, Xinda and
Lu, Junting and
Lin, Qingwei and
Rajmohan, Saravan and
Zhang, Dongmei",
editor = "Lee, Hung-Yi and
Dev, Sunipa and
Li, Jessy and
Elliott, Desmond",
booktitle = "Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing",
month = oct,
year = "2026",
address = "Budapest, Hungary",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2026.emnlp-main.1967/",
pages = "44793--44807",
ISBN = "979-8-89176-459-0",
abstract = "Reinforcement learning with verifiable or outcome-level rewards has recently become an effective post-training paradigm for improving the reasoning abilities of large language models, with GRPO-style optimization achieving strong results on mathematical and other verifiable tasks. However, for multi-step reasoning, outcome-level supervision remains intrinsically sparse: a trajectory that contains partially correct intermediate derivations but makes a late mistake is still treated as a complete failure. As a result, useful reasoning prefixes are penalized together with erroneous continuations, and informative failed trajectories are underutilized. PRMs provide denser step-level feedback, but they often rely on costly process annotation or auxiliary process verifiers, and optimizing local process rewards alone may fail to preserve the global objective of final correctness, leaving room for reward misspecification and reward hacking during reinforcement fine-tuning. We propose PRSA(Process Rewarding via Solution-Space Alignment), a simple framework that bridges outcome supervision and process supervision without requiring human step-level labels. PRSA automatically extracts intermediate-value steps from model outputs and reference solutions, aligns them within the solution space under equivalence and order constraints to estimate reasoning progress, and uses this signal for both reward shaping and token-level prefix-suffix credit assignment. In this way, PRSA preserves the global anchor of final-answer correctness while providing denser and better localized learning signals for partially correct trajectories."
}<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="zhao-etal-2026-process">
<titleInfo>
<title>Process Rewarding via Solution-Space Alignment</title>
</titleInfo>
<name type="personal">
<namePart type="given">ChenZhuo</namePart>
<namePart type="family">Zhao</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Yue</namePart>
<namePart type="family">Huang</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Pu</namePart>
<namePart type="family">Zhao</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Fangkai</namePart>
<namePart type="family">Yang</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Lu</namePart>
<namePart type="family">Wang</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Lin</namePart>
<namePart type="family">Ma</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Zhi</namePart>
<namePart type="family">Yang</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Youling</namePart>
<namePart type="family">Huang</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Xinda</namePart>
<namePart type="family">Wang</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Junting</namePart>
<namePart type="family">Lu</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Qingwei</namePart>
<namePart type="family">Lin</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Saravan</namePart>
<namePart type="family">Rajmohan</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Dongmei</namePart>
<namePart type="family">Zhang</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<originInfo>
<dateIssued>2026-10</dateIssued>
</originInfo>
<typeOfResource>text</typeOfResource>
<relatedItem type="host">
<titleInfo>
<title>Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing</title>
</titleInfo>
<name type="personal">
<namePart type="given">Hung-Yi</namePart>
<namePart type="family">Lee</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Sunipa</namePart>
<namePart type="family">Dev</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Jessy</namePart>
<namePart type="family">Li</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Desmond</namePart>
<namePart type="family">Elliott</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<originInfo>
<publisher>Association for Computational Linguistics</publisher>
<place>
<placeTerm type="text">Budapest, Hungary</placeTerm>
</place>
</originInfo>
<genre authority="marcgt">conference publication</genre>
<identifier type="isbn">979-8-89176-459-0</identifier>
</relatedItem>
<abstract>Reinforcement learning with verifiable or outcome-level rewards has recently become an effective post-training paradigm for improving the reasoning abilities of large language models, with GRPO-style optimization achieving strong results on mathematical and other verifiable tasks. However, for multi-step reasoning, outcome-level supervision remains intrinsically sparse: a trajectory that contains partially correct intermediate derivations but makes a late mistake is still treated as a complete failure. As a result, useful reasoning prefixes are penalized together with erroneous continuations, and informative failed trajectories are underutilized. PRMs provide denser step-level feedback, but they often rely on costly process annotation or auxiliary process verifiers, and optimizing local process rewards alone may fail to preserve the global objective of final correctness, leaving room for reward misspecification and reward hacking during reinforcement fine-tuning. We propose PRSA(Process Rewarding via Solution-Space Alignment), a simple framework that bridges outcome supervision and process supervision without requiring human step-level labels. PRSA automatically extracts intermediate-value steps from model outputs and reference solutions, aligns them within the solution space under equivalence and order constraints to estimate reasoning progress, and uses this signal for both reward shaping and token-level prefix-suffix credit assignment. In this way, PRSA preserves the global anchor of final-answer correctness while providing denser and better localized learning signals for partially correct trajectories.</abstract>
<identifier type="citekey">zhao-etal-2026-process</identifier>
<location>
<url>https://aclanthology.org/2026.emnlp-main.1967/</url>
</location>
<part>
<date>2026-10</date>
<extent unit="page">
<start>44793</start>
<end>44807</end>
</extent>
</part>
</mods>
</modsCollection>
%0 Conference Proceedings
%T Process Rewarding via Solution-Space Alignment
%A Zhao, ChenZhuo
%A Huang, Yue
%A Zhao, Pu
%A Yang, Fangkai
%A Wang, Lu
%A Ma, Lin
%A Yang, Zhi
%A Huang, Youling
%A Wang, Xinda
%A Lu, Junting
%A Lin, Qingwei
%A Rajmohan, Saravan
%A Zhang, Dongmei
%Y Lee, Hung-Yi
%Y Dev, Sunipa
%Y Li, Jessy
%Y Elliott, Desmond
%S Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-459-0
%F zhao-etal-2026-process
%X Reinforcement learning with verifiable or outcome-level rewards has recently become an effective post-training paradigm for improving the reasoning abilities of large language models, with GRPO-style optimization achieving strong results on mathematical and other verifiable tasks. However, for multi-step reasoning, outcome-level supervision remains intrinsically sparse: a trajectory that contains partially correct intermediate derivations but makes a late mistake is still treated as a complete failure. As a result, useful reasoning prefixes are penalized together with erroneous continuations, and informative failed trajectories are underutilized. PRMs provide denser step-level feedback, but they often rely on costly process annotation or auxiliary process verifiers, and optimizing local process rewards alone may fail to preserve the global objective of final correctness, leaving room for reward misspecification and reward hacking during reinforcement fine-tuning. We propose PRSA(Process Rewarding via Solution-Space Alignment), a simple framework that bridges outcome supervision and process supervision without requiring human step-level labels. PRSA automatically extracts intermediate-value steps from model outputs and reference solutions, aligns them within the solution space under equivalence and order constraints to estimate reasoning progress, and uses this signal for both reward shaping and token-level prefix-suffix credit assignment. In this way, PRSA preserves the global anchor of final-answer correctness while providing denser and better localized learning signals for partially correct trajectories.
%U https://aclanthology.org/2026.emnlp-main.1967/
%P 44793-44807
Markdown (Informal)
[Process Rewarding via Solution-Space Alignment](https://aclanthology.org/2026.emnlp-main.1967/) (Zhao et al., EMNLP 2026)
ACL
- ChenZhuo Zhao, Yue Huang, Pu Zhao, Fangkai Yang, Lu Wang, Lin Ma, Zhi Yang, Youling Huang, Xinda Wang, Junting Lu, Qingwei Lin, Saravan Rajmohan, and Dongmei Zhang. 2026. Process Rewarding via Solution-Space Alignment. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 44793–44807, Budapest, Hungary. Association for Computational Linguistics.