Vision Large Language Models (VLLMs) show promise for multi-modal understanding; yet, their application to emotion recognition remains limited by insufficient spatial and contextual awareness and by supervised fine-tuning that uniformly penalizes all reasoning tokens—suppressing valid alternatives. We present Set-of-Vision-Text Rewards (SoVTR), a unified, reward-based framework that guides VLLMs with interpretable and verifiable visual attributes, focusing optimization on the emotion-discriminative reasoning steps that truly determine correctness. SoVTR decomposes supervision into three rewards: (i) Face Reward capturing action units, gaze, and micro-expressions; (ii) Body Reward encoding pose and gesture cues; and (iii) Context Reward modeling person–object relations and scene activities. By combining BERTScore attribute matching with GRPO policy optimization, SoVTR produces attribute-grounded explanation traces while preserving non-critical reasoning diversity. Across DFEW, FERV39k, and ExpW, SoVTR consistently improves both recognition accuracy and grounding quality, delivering substantial gains over strong VLLM baselines.

The rapid advancement of generative AI raises concerns about the misuse of Multimodal LLMs (MLLMs) for large-scale disinformation campaigns on social media. Despite existing research on textual disinformation, a fundamental question remains unanswered: can MLLMs be exploited to fabricate realistic multimodal fake news, and can they reliably detect it? We introduce a multi-agent framework in which a story agent, an image agent, and a critic agent collaborate to produce fake social media posts that plausibly counter true news. We apply the framework to generate over 9,000 paired multimodal news posts across science, health, and entertainment domains, and benchmark 16 open- and closed-source MLLMs for automated detection. We find that most models fall substantially short of human-level accuracy and fail critically on identifying image authenticity. Our research provides a foundation for developing robust defenses against social media fake news. Code and data are available at https://github.com/xiuzhenzhang/Multimodal.

Plane geometry problem solving (PGPS) has recently gained significant attention as a benchmark to assess the multi-modal reasoning capabilities of large vision-language models. Despite the growing interest in PGPS, the research community still lacks a comprehensive overview that systematically synthesizes recent work in PGPS. To fill this gap, we present a survey of existing PGPS studies. We first categorize PGPS methods into an encoder-decoder framework and summarize the corresponding output formats used by their encoders and decoders. Subsequently, we classify and analyze these encoders and decoders according to their architectural designs. Finally, we outline major challenges and promising directions for future research. In particular, we discuss the hallucination issues arising during the encoding phase within encoder-decoder architectures, as well as the problem of data leakage in current PGPS benchmarks.

Predicting next surgical phase, viewed as a form of surgical procedural intent, with high accuracy is essential for robotic real-time surgical systems. Previous approaches struggle with short-term dynamics, occlusions, and phase persistence; end-to-end large vision-language models (LVLMs) can reason from visual inputs but often fail to exploit structured temporal observations. Thus, we introduce SurgAgents, a multi-agent framework for embodied surgical intent prediction. SurgAgents decomposes prediction into specialized agents: (1) a ResNet-based single-frame classifier agent extracts per-frame phase information; (2) a YOLO-based instrument specialist agent extracts per-frame instrument sets; (3) a history descriptor agent composes these facts into frame-level descriptions; (4) a history summarizer agent compresses long histories into a compact temporal narrative; (5) a phase predictor agent assembles the narrative and the current frame into a formatted LVLM prompt; (6) a retrieval agent with a confidence-gated Retrieval-augmented generation (RAG) module selectively retrieves similar case patterns when prediction confidence is low. On Cataract-1K, SurgAgents attains 46.54% macro-F1 with GPT-4o, outperforming the baseline, with systematic ablation studies showing that each agent materially contributes to performance. Latency measurements demonstrate feasibility for robotic real-time surgical support during surgery procedures.

The prevalence of vision-threatening eye diseases is a significant global burden, with many cases remaining undiagnosed or diagnosed too late for effective treatment. Large vision-language models (LVLMs) have the potential to assist in understanding anatomical information, diagnosing eye diseases, and drafting interpretations and follow-up plans, thereby reducing the burden on clinicians and improving access to eye care. However, limited benchmarks are available to assess LVLMs’ performance in ophthalmology-specific applications. In this study, we introduce LMOD, a large-scale multimodal ophthalmology benchmark consisting of 21,993 instances across (1) five ophthalmic imaging modalities: optical coherence tomography, color fundus photographs, scanning laser ophthalmoscopy, lens photographs, and surgical scenes; (2) free-text, demographic, and disease biomarker information; and (3) primary ophthalmology-specific applications such as anatomical information understanding, disease diagnosis, and subgroup analysis. In addition, we benchmarked 13 state-of-the-art LVLM representatives from closed-source, open-source, and medical domains. The results demonstrate a significant performance drop for LVLMs in ophthalmology compared to other domains. Systematic error analysis further identified six major failure modes: misclassification, failure to abstain, inconsistent reasoning, hallucination, assertions without justification, and lack of domain-specific knowledge. In contrast, supervised neural networks specifically trained on these tasks as baselines demonstrated high accuracy. These findings underscore the pressing need for benchmarks in the development and validation of ophthalmology-specific LVLMs.

We introduce GeoDANO, a geometric vision-language model (VLM) with a domain-agnostic vision encoder, for solving plane geometry problems. Although VLMs have been employed for solving geometry problems, their ability to recognize geometric features remains insufficiently analyzed. To address this gap, we propose a benchmark that evaluates the recognition of visual geometric features, including primitives such as dots and lines, and relations such as orthogonality. Our preliminary study shows that vision encoders often used in general-purpose VLMs, e.g., OpenCLIP, fail to detect these features and struggle to generalize across domains. To overcome the limitation, we develop GeoCLIP, a CLIP-based model trained on synthetic geometric diagram–caption pairs. Benchmark results show that GeoCLIP outperforms existing vision encoders in recognizing geometric features. We then propose our VLM, GeoDANO, which augments GeoCLIP with a domain adaptation strategy for unseen diagram styles. GeoDANO outperforms specialized methods for plane geometry problems and GPT-4o on MathVerse. The implementation is available at https://github.com/ml-postech/GeoDANO.

Vision Large Language Models (VLLMs) are transforming the intersection of computer vision and natural language processing; however, the potential of using visual prompts for emotion recognition in these models remains largely unexplored and untapped. Traditional methods in VLLMs struggle with spatial localization and often discard valuable global context. We propose a novel Set-of-Vision prompting (SoV) approach that enhances zero-shot emotion recognition by using spatial information, such as bounding boxes and facial landmarks, to mark targets precisely. SoV improves accuracy in face count and emotion categorization while preserving the enriched image context. Through comprehensive experimentation and analysis of recent commercial or open-source VLLMs, we evaluate the SoV model’s ability to comprehend facial expressions in natural environments. Our findings demonstrate the effectiveness of integrating spatial visual prompts into VLLMs for improving emotion recognition performance.