Recent Multimodal Large Language Models (MLLMs) have made rapid progress in video understanding, demonstrating strong performance across a wide range of perceptual and reasoning tasks. However, their Social Intelligence remains underexplored and difficult to evaluate. Existing benchmarks for social understanding suffer from two major limitations: they primarily evaluate short-horizon interactions and focus largely on explicit perceptual cues, such as facial expressions and speech sentiment. However, real-world social understanding often requires inferring implicit emotions, intentions, beliefs, and interpersonal dynamics from extended contextual interactions rather than isolated expressions or utterances. Consequently, robust evaluation of social intelligence requires long-horizon reasoning over complex social contexts. To address this gap, we introduce SMMU, a Massive Multimodal Social Understanding benchmark. SMMU comprises 264 long-form audiovisual scenes and 1519 question-answer pairs, to systematically evaluate each model across five social dimensions (Emotions, Intent, Knowledge State, Perspective, Relationships) and three cognitive tasks (Comprehension, Reasoning, Prediction). SMMU is designed to require models to integrate contextual evidence across extended interactions in order to interpret, reason about, and anticipate complex human behavior. Evaluating a suite of state-of-the-art MLLMs, we find that even the strongest models achieve only 0.43 Social Understanding Score compared to the average human baseline of 0.85, and 0.33 Social Agency Score compared to the human baseline of 0.78, revealing a substantial gap between current MLLMs and human social intelligence.

Humans acquire knowledge through three cognitive stages: perceiving information, comprehending knowledge, and adapting knowledge to solve novel problems. Videos serve as an effective medium for knowledge acquisition, facilitating a natural progression through these learning stages. However, existing video benchmarks fail to evaluate the knowledge acquisition capabilities of Large Multimodal Models (LMMs). To address this gap, we introduce Video-MMMU, a multi-modal, multi-discipline, multi-track benchmark that evaluates LMMs’ ability to acquire knowledge from college-level, educational videos. Video-MMMU features a collection of 300 videos and 900 human-annotated questions across six disciplines, evaluating knowledge acquisition through stage-aligned question-answer pairs: Perception, Comprehension, and Adaptation. Beyond measuring final accuracy, Video-MMMU proposes the performance gain metric that quantifies an LMM’s learning gain from video, shifting the focus of evaluation from absolute performance to learning efficiency. Our evaluation reveals a substantial gap between human learners and current LMMs, highlighting the need to improve models’ ability to learn and adapt knowledge from video content.

The advances of large foundation models necessitate wide-coverage, low-cost, and zero-contamination benchmarks. Despite continuous exploration of language model evaluations, comprehensive studies on the evaluation of Large Multi-modal Models (LMMs) remain limited. In this work, we introduce LMMS-EVAL, a unified and standardized multimodal benchmark framework with over 50 tasks and more than 10 models to promote transparent and reproducible evaluations. Although LMMS-EVAL offers comprehensive coverage, we find it still falls short in achieving low cost and zero contamination. To approach this evaluation trilemma, we further introduce LMMS-EVAL LITE, a pruned evaluation toolkit that emphasizes both coverage and efficiency. Additionally, we present Multimodal LIVEBENCH that utilizes continuously updating news and online forums to assess models’ generalization abilities in the wild, featuring a low-cost and zero-contamination evaluation approach. In summary, our work highlights the importance of considering the evaluation trilemma and provides practical solutions to navigate the trade-offs in evaluating large multi-modal models, paving the way for more effective and reliable benchmarking of LMMs.