Learning from experience is essential for agents in real-world applications. However, most existing methods rely on static imitation of successful expert trajectories, overlooking the informative signals exposed from failed experiences and lacking a interactive feedback loop with the environment. In this paper, we propose R-IDEAL, a unified framework for insight-driven experience generation and optimization. R-IDEAL first perform contrastive analysis between successful and failed trajectories to automatically synthesize structured insights, consisting of positive plans and negative lessons, which guide agents not only on what to do, but also on what to avoid. These synthesized insights serve as supervision to train a lightweight insight model that generates task-specific guidance. While this supervision stabilizes insight generation, it does not guarantee the effectiveness of the insights due to distribution shifts and execution discrepancies across agents. Therefore, we further optimize the insight model via reinforcement learning using execution feedback from heterogeneous agents in real environments, ensuring that the generated insights effectively improve agent behavior. Experiments on AlfWorld and SciWorld show that R-IDEAL substantially outperforms existing methods, demonstrating strong robustness in out-of-distribution scenarios and serving as a lightweight, plug-and-play module for both closed-source and large-scale models.