Recent advancements have significantly expanded the theoretical context windows of Multimodal Embedding Models (MEMs). However, larger context windows do not necessarily translate into effective comprehension and representation of long-context multimodal inputs, which remains a critical bottleneck for real-world deployment. To address the lack of systematic evaluation in this setting, we introduce *MMLongEmbed*, the first comprehensive benchmark for evaluating MEMs in long-context scenarios. MMLongEmbed comprises four retrieval tasks spanning multiple context-length ranges, covering text, document, and video modalities. Through extensive evaluation of state-of-the-art models, we find that current architectures rely heavily on superficial feature matching and struggle to capture deep semantic and structural dependencies. We further observe that performance degradation varies systematically with context length and key information placement. Moreover, models exhibit substantially different robustness to redundant contextual information across modalities.

With the surge of AI-powered real-world applications, large language models (LLMs) increasingly adopt memory-centric mechanisms to handle multi-turn interactive tasks. Under this paradigm, intermediate processing trajectories, i.e., memory, are central to ensuring LLMs’ faithfulness and effectiveness. Scalable evaluation of the memory management process serves as the cornerstone for further enhancing model performance. Yet, appropriate evaluators for the assessment of memory are currently lacking. Thus, in this work, we seek to employ reward models (RMs) as evaluators and introduce MemoryRewardBench, the first benchmark to systematically study the capability of RMs in assessing LLM memory management processes. MemoryRewardBench covers both long-form comprehension and generation tasks that simulate real-world multi-turn interaction scenarios, featuring 10 distinct settings with different memory management patterns. Evaluations on 13 cutting-edge RMs indicate a diminishing performance gap between open-source and proprietary models, with newer-generation models consistently outperforming their predecessors regardless of parameter count. Additionally, we reveal the fundamental limitations of RMs in evaluating LLMs’ memory.