Reinforcement learning (RL) has emerged as a powerful paradigm for post-training large language model (LLM) agents. However, credit assignment in multi-turn environments remains a challenge. Agents typically receive sparse, trajectory-level rewards only at the end of an episode, making it difficult to identify which specific intermediate actions led to success or failure. Consequently, effectively propagating delayed outcomes back to individual steps without relying on costly auxiliary value models remains an open problem. In this paper, we propose Generalized Advantage Grouped Policy Optimization (GAGPO), a critic-free RL method that enables precise, step-aligned temporal credit assignment. GAGPO constructs a non-parametric grouped value proxy from sampled rollouts to compute TD/GAE-style temporal advantages, recursively propagating outcome supervision backward through time. Coupled with group-wise advantage normalization and an action-level importance ratio, GAGPO extracts stable and localized optimization signals directly from multi-turn trajectories. Experiments on ALFWorld and WebShop demonstrate that GAGPO outperforms strong RL baselines. Further analyses reveal faster early-stage learning, improved interaction efficiency, and smoother optimization dynamics, offering a simple yet highly effective framework for multi-turn agentic RL.