As large language models are increasingly deployed as autonomous agents, optimizing their behavior in agentic reasoning and tool-use environments remains challenging due to the stochasticity of environment trajectories and associated high reward latencies. Existing automated prompt optimization (APO) paradigms struggle in these regimes: discrete evolutionary methods (e.g., GEPA, ACE) lack parametric memory, while instance-level reinforcement learning (RL) approaches suffer from severe gradient variance and reward collapse on small, human-curated datasets. To address these limitations, we introduce COMPASS (Critique-guided Optimization of MetaPolicies for Agentic SystemS), a sample-efficient dual-agent framework that shifts optimization toward task-level metapolicies and demonstrates that an RL-based prompter policy can be effectively trained on such tasks. To overcome sparse feedback, COMPASS introduces a Critique Experience Buffer that couples scalar rewards with dense textual critiques. This mechanism allows an active “Prompter” policy to internalize corrective linguistic patterns and amortize experiential self-reflection directly into its weights. Empirical evaluations demonstrate that COMPASS stabilizes training and yields up to 2.4× faster convergence. On complex reasoning (BBEH) and tool-use (𝜏-bench) benchmarks, our framework improves success rates from 55% to 90% and 74% to 91%, respectively, while naturally discovering specialized algorithmic strategies.