With the surge of AI-powered real-world applications, large language models (LLMs) increasingly adopt memory-centric mechanisms to handle multi-turn interactive tasks. Under this paradigm, intermediate processing trajectories, i.e., memory, are central to ensuring LLMs’ faithfulness and effectiveness. Scalable evaluation of the memory management process serves as the cornerstone for further enhancing model performance. Yet, appropriate evaluators for the assessment of memory are currently lacking. Thus, in this work, we seek to employ reward models (RMs) as evaluators and introduce MemoryRewardBench, the first benchmark to systematically study the capability of RMs in assessing LLM memory management processes. MemoryRewardBench covers both long-form comprehension and generation tasks that simulate real-world multi-turn interaction scenarios, featuring 10 distinct settings with different memory management patterns. Evaluations on 13 cutting-edge RMs indicate a diminishing performance gap between open-source and proprietary models, with newer-generation models consistently outperforming their predecessors regardless of parameter count. Additionally, we reveal the fundamental limitations of RMs in evaluating LLMs’ memory.

Long-context models(LCMs) have witnessed remarkable advancements in recent years, facilitating real-world tasks like long-document QA. The success of LCMs is founded on the hypothesis that the model demonstrates strong fidelity, enabling it to respond based on the provided long context rather than relying solely on the intrinsic knowledge acquired during pre-training. Yet, in this paper, we find that open-sourced LCMs are not as faithful as expected. We introduce L-CiteEval, an out-of-the-box suite that can assess both generation quality and fidelity in long-context understanding tasks. It covers 11 tasks with context lengths ranging from 8K to 48K and a corresponding automatic evaluation pipeline. Evaluation of 11 cutting-edge closed-source and open-source LCMs indicates that, while there are minor differences in their generation, open-source models significantly lag behind closed-source counterparts in terms of fidelity. Furthermore, we analyze the benefits of citation generation for LCMs from both the perspective of explicit model output and the internal attention mechanism.