Learning from experience is essential for agents in real-world applications. However, most existing methods rely on static imitation of successful expert trajectories, overlooking the informative signals exposed from failed experiences and lacking a interactive feedback loop with the environment. In this paper, we propose R-IDEAL, a unified framework for insight-driven experience generation and optimization. R-IDEAL first perform contrastive analysis between successful and failed trajectories to automatically synthesize structured insights, consisting of positive plans and negative lessons, which guide agents not only on what to do, but also on what to avoid. These synthesized insights serve as supervision to train a lightweight insight model that generates task-specific guidance. While this supervision stabilizes insight generation, it does not guarantee the effectiveness of the insights due to distribution shifts and execution discrepancies across agents. Therefore, we further optimize the insight model via reinforcement learning using execution feedback from heterogeneous agents in real environments, ensuring that the generated insights effectively improve agent behavior. Experiments on AlfWorld and SciWorld show that R-IDEAL substantially outperforms existing methods, demonstrating strong robustness in out-of-distribution scenarios and serving as a lightweight, plug-and-play module for both closed-source and large-scale models.

Language models pre-trained on general text have achieved impressive results in diverse fields. Yet, the distinct linguistic characteristics of task-oriented dialogues (TOD) compared to general text limit the practical utility of existing language models. Current task-oriented dialogue pre-training methods overlook the one-to-many property of conversations, where multiple responses can be appropriate given the same conversation context.In this paper, we propose a novel dialogue pre-training model called DivTOD, which collaborates with LLMs to learn diverse task-oriented dialogue representations. DivTOD guides LLMs in transferring diverse knowledge to smaller models while removing domain knowledge that contradicts task-oriented dialogues. Experiments show that our model outperforms strong TOD baselines on various downstream dialogue tasks and learns the intrinsic diversity of task-oriented dialogues.