Localizing issue-relevant code regions is a critical step in automated software engineering. However, due to their reliance on sparse trajectory-level signals, existing methods cannot identify which per-turn actions matter and often discover correct code regions during exploration but fail to commit them. To address these limitations, we propose an action-aware reinforcement learning method that combines a *per-turn reward sequence* rewarding both the discovery and commitment of gold code regions with an *action-level advantage estimation* scheme that isolates each action’s credit by grouping turns sharing the same exploration context. Extensive evaluations on two benchmarks show that our method achieves 1.58% and 8.55% improvements over state-of-the-art in average F1, with our 4B model outperforming baselines up to 8× larger. Our code is available at https://anonymous.4open.science/r/A2Agent.