Xinhao Yao
Author directory2026
Skip-Connected Rollouts for Mid-Trajectory Rewards in GRPO
Fengwei Teng | Jinyi Bai | Xinhao Yao | Demi Wang | Jiahao Zhao | Zhijiang Guo
Findings of the Association for Computational Linguistics: EMNLP 2026
Fengwei Teng | Jinyi Bai | Xinhao Yao | Demi Wang | Jiahao Zhao | Zhijiang Guo
Findings of the Association for Computational Linguistics: EMNLP 2026
Group Relative Policy Optimization (GRPO) and its variants have proven effective for reinforcement learning with verifiable rewards through outcome-based feedback. Incorporating intermediate credit, however, remains challenging. We show that, under practical sampling budgets, token-level Monte Carlo estimates at early reasoning positions have high variance, which can cause frequent advantage-sign errors after group-relative normalization and yield wrong-direction token-level gradient contributions. At later positions, these estimates increasingly resemble the terminal outcome reward and provide limited additional information.
We propose SCRoll, a rollout-side module that augments group-relative policy optimizers with randomized single-boundary segment-level rewards while leaving the downstream optimizer interface unchanged. SCRoll uses a skip-connected two-stream rollout. An upstream stream generates an early-stopped reasoning segment s, while a downstream stream samples G continuations conditioned on [s,q], which places the segment before the original problem. This construction retains the segment as preceding reasoning context while mitigating the reward homogenization associated with pure-prefix continuation. The same downstream group supplies terminal outcomes for the standard group-relative objective and a Monte Carlo aggregate used as the upstream segment reward, avoiding a separate objective-specific rollout group. The upstream segment is optimized with a sample-efficient single-stream objective.
Across Qwen2.5-Math-7B and Llama-3.2-3B-Instruct, pairing SCRoll with four representative group-relative optimizers improves performance in most evaluated settings. The clearest gains occur with CISPO, with smaller positive average gains for GRPO and DAPO and marginal gains for GSPO. A decoupled two-process baseline can achieve comparable performance with substantially more generated tokens, while SCRoll performs better under a matched generated-token budget, supporting the sample-efficiency benefit of rollout reuse.
We propose SCRoll, a rollout-side module that augments group-relative policy optimizers with randomized single-boundary segment-level rewards while leaving the downstream optimizer interface unchanged. SCRoll uses a skip-connected two-stream rollout. An upstream stream generates an early-stopped reasoning segment s, while a downstream stream samples G continuations conditioned on [s,q], which places the segment before the original problem. This construction retains the segment as preceding reasoning context while mitigating the reward homogenization associated with pure-prefix continuation. The same downstream group supplies terminal outcomes for the standard group-relative objective and a Monte Carlo aggregate used as the upstream segment reward, avoiding a separate objective-specific rollout group. The upstream segment is optimized with a sample-efficient single-stream objective.
Across Qwen2.5-Math-7B and Llama-3.2-3B-Instruct, pairing SCRoll with four representative group-relative optimizers improves performance in most evaluated settings. The clearest gains occur with CISPO, with smaller positive average gains for GRPO and DAPO and marginal gains for GSPO. A decoupled two-process baseline can achieve comparable performance with substantially more generated tokens, while SCRoll performs better under a matched generated-token budget, supporting the sample-efficiency benefit of rollout reuse.
UniGeM: Unifying Data Selection and Mixing via Geometric Exploration and Mining
Changhao Wang | Yunfeiyu | Xinhao Yao | Jiaolong Yang | Lu Yu | Junpeng Fang | Chaobo Li | Riccardo Cantoro | Qing Cui | Jun Zhou
Findings of the Association for Computational Linguistics: ACL 2026
Changhao Wang | Yunfeiyu | Xinhao Yao | Jiaolong Yang | Lu Yu | Junpeng Fang | Chaobo Li | Riccardo Cantoro | Qing Cui | Jun Zhou
Findings of the Association for Computational Linguistics: ACL 2026
The scaling of Large Language Models (LLMs) is increasingly limited by data quality. Most methods handle data mixing and sample selection separately, which can break the structure in code corpora. We introduce UniGeM, a framework that unifies mixing and selection by treating data curation as a manifold approximation problem without training proxy models or relying on external reference datasets. UniGeM operates hierarchically: Macro-Exploration learns mixing weights with stability-based clustering; Micro-Mining filters high-quality instances by their geometric distribution to ensure logical consistency. Validated by training 8B and 16B MoE models on 100B tokens, UniGeM achieves 2.0 × data efficiency over a random baseline and further improves overall performance compared to SOTA methods in reasoning-heavy evaluations and multilingual generalization.