Long-horizon LLM agents often fail from procedural errors such as forgetting constraints, misordering tool calls, or stopping prematurely. Because these slips are unpredictable, the same agent can succeed on one run and fail on an identical task later. The resulting execution trajectories contain rich signals about which procedural choices lead to success or failure, yet existing approaches that extract reusable guidance from agent trajectories either introduce noise through imprecise distillation or depend on external retrievers for deployment. We propose SkillForge, a training-free framework that rethinks both the construction and deployment of procedural guidance: it pairs successful and failed attempts on the same task to identify critical procedural choices, consolidates and filters candidate guidance to remove low-quality entries, and loads relevant guidance on demand at inference time only when the agent’s current context signals its relevance. Across agent benchmarks including 𝜏²-Bench and Terminal-Bench, SkillForge improves repeated-run consistency, reducing the consistency gap by up to 37% and lowering total token usage by up to 23.6% compared to baselines, while moderately improving single-trial performance. These results suggest a practical path toward distilling high-quality procedural guidance from past trajectories for more consistent and efficient long-horizon agents.

Long-horizon tasks that require sustained reasoning and multiple tool interactions remain challenging for LLM agents: small errors compound across steps, and even state-of-the-art models often hallucinate or lose coherence. We identify context management as the central bottleneck—extended histories cause agents to overlook critical evidence or become distracted by irrelevant information, thus failing to replan or reflect from previous mistakes. To address this, we propose COMPASS (Context-Organized Multi-Agent Planning and Strategy System), a lightweight hierarchical framework that separates tactical execution, strategic oversight, and context organization into three specialized components: (1) a Main Agent that performs reasoning and tool use, (2) a Meta-Thinker that monitors progress and issues strategic interventions, and (3) a Context Manager that maintains concise, relevant progress briefs for different reasoning stages. Across three challenging benchmarks—GAIA, BrowseComp, and Humanity’s Last Exam—COMPASS improves accuracy by up to 20% relative to both single- and multi-agent baselines. We further introduce a test-time scaling extension that elevates performance to match established DeepResearch agents, and a post-training pipeline that delegates context management to smaller models for enhanced efficiency.

Recently, decomposing complex problems into simple subtasks–a crucial part of human-like natural planning–to solve the given problem has significantly boosted the performance of large language models (LLMs). However, leveraging such planning structures during post-training to boost the performance of smaller open-source LLMs remains underexplored. Motivated by this, we introduce PLAN-TUNING, a unified post-training framework that (i) distills synthetic task decompositions (termed “planning trajectories”) from large-scale LLMs and (ii) fine-tunes smaller models via supervised and reinforcement-learning objectives designed to mimic these planning processes to improve complex reasoning. On GSM8k and the MATH benchmarks, plan-tuned models outperform strong baselines by an average ~7%. Furthermore, plan-tuned models show better generalization capabilities on out-of-domain datasets, with average ~10% and ~12% performance improvements on OlympiadBench and AIME 2024, respectively. Our detailed analysis demonstrates how planning trajectories improves complex reasoning capabilities, showing that PLAN-TUNING is an effective strategy for improving task-specific performance of smaller LLMs.