As large language models are increasingly deployed as autonomous agents, optimizing their behavior in agentic reasoning and tool-use environments remains challenging due to the stochasticity of environment trajectories and associated high reward latencies. Existing automated prompt optimization (APO) paradigms struggle in these regimes: discrete evolutionary methods (e.g., GEPA, ACE) lack parametric memory, while instance-level reinforcement learning (RL) approaches suffer from severe gradient variance and reward collapse on small, human-curated datasets. To address these limitations, we introduce COMPASS (Critique-guided Optimization of MetaPolicies for Agentic SystemS), a sample-efficient dual-agent framework that shifts optimization toward task-level metapolicies and demonstrates that an RL-based prompter policy can be effectively trained on such tasks. To overcome sparse feedback, COMPASS introduces a Critique Experience Buffer that couples scalar rewards with dense textual critiques. This mechanism allows an active “Prompter” policy to internalize corrective linguistic patterns and amortize experiential self-reflection directly into its weights. Empirical evaluations demonstrate that COMPASS stabilizes training and yields up to 2.4× faster convergence. On complex reasoning (BBEH) and tool-use (𝜏-bench) benchmarks, our framework improves success rates from 55% to 90% and 74% to 91%, respectively, while naturally discovering specialized algorithmic strategies.

Modeling long user histories plays a pivotal role in enhancing recommendation systems, allowing to capture users’ evolving preferences, resulting in more precise and personalized recommendations. In this study, we tackle the challenges of modeling long user histories for preference understanding in natural language. Specifically, we introduce a new User Embedding Module (UEM) that efficiently processes user history in free-form text by compressing and representing them as embeddings, to use them as soft prompts to a language model (LM). Our experiments demonstrate the superior capability of this approach in handling significantly longer histories compared to conventional text-based methods, yielding substantial improvements in predictive performance. Models trained using our approach exhibit substantial enhancements, with up to 0.21 and 0.25 F1 points improvement over the text-based prompting baselines. The main contribution of this research is to demonstrate the ability to bias language models via user signals.