Large language model (LLM)-based agents solve complex long-horizon tasks through iterative interaction with environments, where effectively learning from experience under sparse or delayed feedback remains a key challenge. Existing approaches typically rely on either non-parametric memory-based experience reusing or parametric reinforcement learning, each with inherent limitations when used in isolation. To this end, we propose **ExpRL**, a unified experience learning framework that bridges non-parametric memory and parametric policy optimization for LLM agents. ExpRL induces abstracted success and failure experiences from on-policy interaction trajectories, stores them in an external memory, and integrates experience acquisition and utilization directly into the reinforcement learning process via an agentic experience acquisition mechanism. This design enables memory and policy to co-evolve and mutually reinforce each other, effectively unifying online policy learning with offline experience reuse. Experiments on ALFWorld and WebShop demonstrate that ExpRL achieves an average performance improvement of **13.0** over GRPO, with improved learning efficiency and stability. We further show that the induced experience memory generalizes across different models and can be directly transferred to stronger LLMs without additional training.

Stickers, while widely recognized for enhancing empathetic communication in online interactions, remain underexplored in current empathetic dialogue research, notably due to the challenge of a lack of comprehensive datasets. In this paper, we introduce the Agent for STICKERCONV (Agent4SC), which uses collaborative agent interactions to realistically simulate human behavior with sticker usage, thereby enhancing multimodal empathetic communication. Building on this foundation, we develop a multimodal empathetic dialogue dataset, STICKERCONV, comprising 12.9K dialogue sessions, 5.8K unique stickers, and 2K diverse conversational scenarios. This dataset serves as a benchmark for multimodal empathetic generation. To advance further, we propose PErceive and Generate Stickers (PEGS), a multimodal empathetic response generation framework, complemented by a comprehensive set of empathy evaluation metrics based on LLM. Our experiments demonstrate PEGS’s effectiveness in generating contextually relevant and emotionally resonant multimodal empathetic responses, contributing to the advancement of more nuanced and engaging empathetic dialogue systems.