Group Relative Policy Optimization (GRPO) and its variants have proven effective for reinforcement learning with verifiable rewards through outcome-based feedback. Incorporating intermediate credit, however, remains challenging. We show that, under practical sampling budgets, token-level Monte Carlo estimates at early reasoning positions have high variance, which can cause frequent advantage-sign errors after group-relative normalization and yield wrong-direction token-level gradient contributions. At later positions, these estimates increasingly resemble the terminal outcome reward and provide limited additional information.

We propose SCRoll, a rollout-side module that augments group-relative policy optimizers with randomized single-boundary segment-level rewards while leaving the downstream optimizer interface unchanged. SCRoll uses a skip-connected two-stream rollout. An upstream stream generates an early-stopped reasoning segment s, while a downstream stream samples G continuations conditioned on [s,q], which places the segment before the original problem. This construction retains the segment as preceding reasoning context while mitigating the reward homogenization associated with pure-prefix continuation. The same downstream group supplies terminal outcomes for the standard group-relative objective and a Monte Carlo aggregate used as the upstream segment reward, avoiding a separate objective-specific rollout group. The upstream segment is optimized with a sample-efficient single-stream objective.

Across Qwen2.5-Math-7B and Llama-3.2-3B-Instruct, pairing SCRoll with four representative group-relative optimizers improves performance in most evaluated settings. The clearest gains occur with CISPO, with smaller positive average gains for GRPO and DAPO and marginal gains for GSPO. A decoupled two-process baseline can achieve comparable performance with substantially more generated tokens, while SCRoll performs better under a matched generated-token budget, supporting the sample-efficiency benefit of rollout reuse.

Well-designed prompts are crucial for enhancing Large language models’ (LLMs) reasoning capabilities while aligning their outputs with task requirements across diverse domains. However, manually designed prompts require expertise and iterative experimentation. While existing prompt optimization methods aim to automate this process, they rely heavily on external references such as ground truth or by humans, limiting their applicability in real-world scenarios where such data is unavailable or costly to obtain. To address this, we propose Self-Supervised Prompt Optimization (SPO), a cost-efficient framework that discovers effective prompts for both closed and open-ended tasks without requiring external reference. Motivated by the observations that prompt quality manifests directly in LLM outputs and LLMs can effectively assess adherence to task requirements, we derive evaluation and optimization signals purely from output comparisons. Specifically, SPO selects superior prompts through pairwise output comparisons evaluated by an LLM evaluator, followed by an LLM optimizer that aligns outputs with task requirements. Extensive experiments demonstrate that SPO outperforms state-of-the-art prompt optimization methods, achieving comparable or superior results with significantly lower costs (e.g., 1.1% to 5.6% of existing methods) and fewer samples (e.g., three samples).