This paper studies retrieval-augmented generation (RAG), which leverages external knowledge to improve the performance of language models. Despite the remarkable progress, current RAG approaches are still far from satisfactory due to inadequate retrieval and potential hallucinations. Towards this end, we propose a novel approach named Task Refinement with Individual Optimization (TRIO) for RAG. The core idea of our TRIO is to equip three well-designed agents with an iterative refinement paradigm by aligning their individual optimization objectives. Specifically, we first expand ambiguous queries with a query extension agent and quantify the reward based on the improved answer quality, which can be utilized to refine our agent for higher expansion efficiency. To enhance the retrieval quality, we include a unanimity voting to annotate the expanded query as insufficient or sufficient, and their generative outcomes are utilized as ground truth to optimize the judge agent. To further improve evidence utilization and answer accuracy, an answer agent is optimized with dynamic matching-based rewards with curriculum learning for final outputs. Extensive experiments across multiple benchmark datasets validate the effectiveness of the proposed TRIO in comparison with state-of-the-art approaches. Our code is available at https://github.com/yingtaoren/TRIO.

Multi-step retrieval-augmented generation has attracted increasing attention due to its capacity to improve the factuality of large language models with iterative retrieved knowledge. However, the performance of multi-step RAG systems is susceptible to potential retrieval noise and fabricated documents in real-world scenarios. Current approaches usually utilize supervised fine-tuning on predetermined noisy contexts to enhance the robustness. However, their performance remains inadequate when it comes to more complicated long-context scenarios due to the lack of adaptability. Towards this end, we propose a novel framework named Context-attended Adversarial Reinforcement Learning (CARE) for multi-step RAG systems against attacks. The core of our CARE is to conduct reinforcement learning on adversarial samples which are alternatingly enhanced with text gradients. In particular, our CARE includes a reward model to identify the accuracy of responses, which is minimized for the generation of adversarial samples with text gradients. These context-attended noisy samples are then utilized for reinforcement learning to maximize the rewards. The whole framework is conducted alternatingly from easy to hard samples to ensure the smoothness of the optimization. Extensive experiments on multi-step RAG benchmark datasets are conducted to validate the superiority of our proposed CARE in multiple noisy scenarios. Our code is available at https://github.com/yingtaoren/CARE.