Reinforcement learning (RL) for long-horizon agents faces two key challenges. First, parallel linear rollouts tend to converge toward homogeneous reasoning patterns, leading to cognitive-mode collapse. Second, existing methods refine credit assignment along only a single dimension, making it difficult to simultaneously achieve state-aligned comparison and meta-reasoning discrimination. We propose STARK (Strategic and Tactical Advantage Reinforcement via Confidence Guided Keypoint Exploration) to resolve both issues. STARK performs corrective exploration through confidence-driven branching at uncertain decision points and injects reflection into new branches. To empower these keypoint-expanded trajectories, STARK introduces a dual-track strategic and tactical advantage estimator: The strategic track compares reasoning modes under shared environmental observations, while the tactical track evaluates execution quality within each meta-reasoning tag group. Extensive experiments on ALFWorld, ScienceWorld, and WebShop show that STARK consistently outperforms strong baselines across model scales and task domains, and enables a 1.5B model to surpass GPT-5 on the out-of-domain settings.

Reinforcement learning has empowered large language models to act as intelligent agents, yet training them for long-horizon tasks remains challenging due to the scarcity of high-quality trajectories, especially under limited resources. Existing methods typically scale up rollout sizes and indiscriminately allocate computational resources among intermediate steps. Such attempts inherently waste substantial computation budget on trivial steps while failing to guarantee sample quality. To address this, we propose SPARK (Strategic Policy-Aware exploRation via Key-state dynamic branching), a novel framework that selectively branches at critical decision states for resource-efficient exploration. Our key insight is to activate adaptive branching exploration at critical decision points to probe promising trajectories, thereby achieving precise resource allocation that prioritizes sampling quality over blind coverage. This design leverages the agent’s intrinsic decision-making signals to reduce dependence on human priors, enabling the agent to autonomously expand exploration and achieve stronger generalization. Experiments across diverse tasks (e.g., embodied planning), demonstrate that SPARK achieves superior success rates with significantly fewer training samples, exhibiting robust generalization even in unseen scenarios. Our code and checkpoints are available at https://github.com/jinyangwu/SPARK.