Huaibing Xie

Author directory

2026

Group-relative policy optimization (GRPO) provides a stable recipe for outcome-based RL on language model reasoning, but its credit assignment is coarse: it broadcasts a single trajectory advantage to every token, regardless of how individual steps contribute to the outcome. We call this the uniform-broadcast failure mode and address it with RLPgR, a dependency-aware rescaling of the existing GRPO advantage. RLPgR parses each rollout into a Progress Dependency Graph (PDG), where each node groups a contiguous span of steps into a milestone-level unit with a locally validatable artifact, and directed edges connect a node to the artifacts it depends on. Using the PDG, RLPgR rescales the group-relative advantage by a small, bounded, sign-preserving factor at the node level. This preserves the outcome-governed rollout direction while each step’s update better reflects its dependency role, avoiding the reward-hacking drift of densely shaped process rewards. To reduce PDG construction cost, we train an open Qwen3-8B progress annotator amortizing a one-time proprietary bootstrap. Across model families and sizes on five math reasoning benchmarks, RLPgR is competitive with strong outcome-RL and process-RL baselines under matched training budget. Three controlled studies (scaling-strength sensitivity, segment-level baselines, and PDG label perturbation) support the view that the gain comes from structured credit assignment rather than incidental tuning.
Long-context reasoning remains a critical challenge for Large Language Models (LLMs). Current research typically focuses on curating specialized training data by treating long-context reasoning as a monolithic task, yet it is challenging to verify its quality due to its internal cognitive complexity. In this paper, we propose a paradigm shift: viewing long-context reasoning as a hierarchical spectrum of core atomic skills. To operationalize this, we present an Anchor-based Reasoning (AbR) pipeline that automatically synthesizes strictly verifiable training data for each skill, thereby enabling reliable quality. Diagnostic analyses confirm that proficiency in these atomic skills strongly predicts general long-context reasoning performance. By training exclusively on a highly compact set of merely four thousand synthesized samples with Reinforcement Learning (RL), we explicitly sharpen these foundational capabilities. Extensive experiments across six challenging benchmarks demonstrate that this targeted RL causally delivers robust and generalizable performance gains, outperforming DeepSeek-R1-distill-32B by an average margin of 7.7% (improving from 46.3% to 54.0%). Furthermore, in-depth ablations reveal that these skills operate synergistically, establishing a scalable, data-efficient, and verifiable path to advancing long-context reasoning.