Group Relative Policy Optimization (GRPO) and its variants have proven effective for reinforcement learning with verifiable rewards through outcome-based feedback. Incorporating intermediate credit, however, remains challenging. We show that, under practical sampling budgets, token-level Monte Carlo estimates at early reasoning positions have high variance, which can cause frequent advantage-sign errors after group-relative normalization and yield wrong-direction token-level gradient contributions. At later positions, these estimates increasingly resemble the terminal outcome reward and provide limited additional information.

We propose SCRoll, a rollout-side module that augments group-relative policy optimizers with randomized single-boundary segment-level rewards while leaving the downstream optimizer interface unchanged. SCRoll uses a skip-connected two-stream rollout. An upstream stream generates an early-stopped reasoning segment s, while a downstream stream samples G continuations conditioned on [s,q], which places the segment before the original problem. This construction retains the segment as preceding reasoning context while mitigating the reward homogenization associated with pure-prefix continuation. The same downstream group supplies terminal outcomes for the standard group-relative objective and a Monte Carlo aggregate used as the upstream segment reward, avoiding a separate objective-specific rollout group. The upstream segment is optimized with a sample-efficient single-stream objective.

Across Qwen2.5-Math-7B and Llama-3.2-3B-Instruct, pairing SCRoll with four representative group-relative optimizers improves performance in most evaluated settings. The clearest gains occur with CISPO, with smaller positive average gains for GRPO and DAPO and marginal gains for GSPO. A decoupled two-process baseline can achieve comparable performance with substantially more generated tokens, while SCRoll performs better under a matched generated-token budget, supporting the sample-efficiency benefit of rollout reuse.

Large Language Models (LLMs) are powerful but prone to hallucinations due to static knowledge. Retrieval-Augmented Generation (RAG) helps by injecting external information, but current methods often are costly, generalize poorly, or ignore the model’s internal knowledge. In this paper, we introduce Smart-Searcher, a novel framework designed to train LLMs to adaptively leverage both internal and external knowledge sources. Smart-Searcher employs a two-stage training strategy: an initial SFT Cold-start phase for preliminary format learning, followed by RL for Dynamic Knowledge Acquisition. The RL stage uses outcome-supervision to encourage exploration, incorporates a reward mechanism for internal knowledge utilization, and integrates a memorization mechanism to continuously assimilate retrieved information, thereby enriching the model’s internal knowledge. By leveraging internal knowledge and external search engine, the model continuously improves its capabilities, enabling efficient retrieval-augmented reasoning. Our experiments demonstrate that Smart-Searcher outperforms previous RAG and reasoning methods and achieves efficient retrieval. The code is available at https://github.com/RUCAIBox/R1-Searcher-plus.