Reinforcement learning for long-horizon agents and multi-step reasoning models typically relies on outcome-level rewards, which provide only trajectory-level supervision and make it difficult to identify the contribution of individual intermediate actions. As a result, redundant actions in successful trajectories may be over-reinforced, while useful actions in failed trajectories may be undesirably penalized. To address this issue, we propose **Dual Experience Pool Policy Optimization (DEPPO)**, a critic-free and experience-driven framework for adaptive step-level credit assignment. DEPPO maintains two structured experience pools for successful and failed trajectories, and estimates the relative quality of each action by comparing its state-conditional likelihood under the two pools. The resulting success–failure preference signal is used to adaptively rescale the advantage, amplifying actions aligned with successful patterns while suppressing those associated with failure modes. Furthermore, DEPPO introduces an asymmetric and dynamically evolving update mechanism: the success pool emphasizes both correctness and novelty through freshness-aware insertion, while the failure pool focuses on consistently risky behaviors, with decay and pruning to keep the memory compact and up-to-date. Without additional value models or process-level supervision, DEPPO provides a lightweight, interpretable, and effective solution for fine-grained credit assignment in long-horizon reinforcement learning. Code available at: https://anonymous.4open.science/r/DEPPO-1051/